tag:blogger.com,1999:blog-44431035632458228572024-03-12T20:10:20.054-07:00Stuff I doFábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-4443103563245822857.post-13578040738974556892019-11-09T06:24:00.003-08:002019-11-09T06:24:33.648-08:00From Scratch: building a Time Domain ReflectometerTDR or Time Domain Reflectometry is a method for measuring transmission lines - cables and alike - used for the transmission of high-speed signals.<br />
<br />
TDR can be used to check the length of the cable, can detect the presence and the place of a short, break, and other errors.<br />
<br />
First, let's take a look at what we're going to build and what we can do with it.<br />
<br />
<a name='more'></a><h2>
<b>Transmission line</b></h2>
<div>
A transmission line is a specific cable or other structure that's used to conduct high-frequency AC current. A transmission line might carry voice, provide AC mains to your home or carry very low voltage radio frequency signals.</div>
<div>
<br /></div>
<div>
We mostly care about radio frequencies of small or moderate power, so "transmisson line" will mostly mean coaxial (unbalanced) cables, mostly 50 Ohm RG58, H155, H1000, etc. The devices in this article can be modified to work with cables of various characteristic impedance and balance. (Worst case you might need to use a transformer.)</div>
<h2>
<b>Velocity factor </b></h2>
TDR can be used to measuring feed line (cable) length or velocity factor. Velocity factor is a number given in percents or in fractions of one, and it is the ratio of the signal propagation speed in the transmission line to the speed of light in vacuum. A velocity factor of 66% or 0.66 means the signal propagates at 0.66 * 300,000 km/s = 198,000 km/s.<br />
<h2>
<b>The method</b></h2>
If a signal is sent along a cable, it will (partially) reflect from the end unless it is terminated with the characteristic impedance. Receiving the reflected signal and measuring the time of flight makes possible to calculate the cable length OR the velocity factor. Knowing one is needed to calculate the other.<br />
<br />
We're going to use short, sharp pulses. Such pulses can be generated with cheap transistors working in <i>avalanche mode</i>, and can be read using cheap oscilloscopes (relatively speaking). To make the transistors work well in avalanche mode, we're going to need a voltage well in excess of one hundred volts.<br />
<h2>
<b>The setup</b></h2>
The measurement setup consists of 3 devices:<b></b><br />
<ul>
<li>An oscilloscope. An old analog or a modern (but cheap) digital one will suffice. 100MHz of bandwidth is probably the minimum, 200MHz is recommended. More will not hurt, but faster scopes get expensive quickly. It must either have a 50 Ohm input or must be used with proper termination.</li>
<li>A high voltage generator with about 200V output (it needs to supply little current).</li>
<li>An avalanche transistor pulse generator. Sounds much more complicated than it really is.</li>
</ul>
<h2>
<b>The scope</b></h2>
I'm going to use a Hantek DSO5202P scope:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtT9eZZywCJjBpNaPN1gIgtwEAt6_1gKtlzltu6v8fKDP9AW28ah_BItI2Jg6Cig1oUfV_xmu0gr9Ec9XU7If7kv4xuzAcJOwysjFjbLkw5ZkdqE8DtsLFE6Jyva7u-tf8NE4-sllikk0/s1600/hantek-dso5202p.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="384" data-original-width="786" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtT9eZZywCJjBpNaPN1gIgtwEAt6_1gKtlzltu6v8fKDP9AW28ah_BItI2Jg6Cig1oUfV_xmu0gr9Ec9XU7If7kv4xuzAcJOwysjFjbLkw5ZkdqE8DtsLFE6Jyva7u-tf8NE4-sllikk0/s640/hantek-dso5202p.jpg" width="640" /></a></div>
<br />
This is how a pulse and it's reflection looks like on the screen:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_NiHp8Y6krsBOaumqvcIP6x4TCNQFl6l-2-qCZX18gcvSY6gBfaLlhkEL4lVJKQduVlKxs8TAuFn3dkiCFyWw1vf2zugf1d_sAK9WjY_b2_BP1NccvQLWtM4pUg9rmnJvXzZIXidcdNg/s1600/pic_7_3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="800" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_NiHp8Y6krsBOaumqvcIP6x4TCNQFl6l-2-qCZX18gcvSY6gBfaLlhkEL4lVJKQduVlKxs8TAuFn3dkiCFyWw1vf2zugf1d_sAK9WjY_b2_BP1NccvQLWtM4pUg9rmnJvXzZIXidcdNg/s640/pic_7_3.png" width="640" /></a></div>
<br />
<br />
An Old HP 1745A will work just fine:<br />
<b></b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjghrbLnBlEpmRUleOlIw19B-9eicu3brzmivWou_E5qvOcj10qdjMYhQG59p8pnqP1lrjIqI7uuWJXWGVNJKYjCEpB0-YymoEXOxgNWK5ydNDQFLx32qWYZON0VAmIwIJ1Dp2PTLSoTn8/s1600/szkop_elol.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjghrbLnBlEpmRUleOlIw19B-9eicu3brzmivWou_E5qvOcj10qdjMYhQG59p8pnqP1lrjIqI7uuWJXWGVNJKYjCEpB0-YymoEXOxgNWK5ydNDQFLx32qWYZON0VAmIwIJ1Dp2PTLSoTn8/s640/szkop_elol.JPG" width="640" /></a></div>
<br />
I think this is prettier :) Pulse and reflection without termination:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEih2u2HLXcTOQ21e49bePmPufDbJZXa91NjkyX91jo9eAEr9VF2wh7MGm3IKJW5b1UwBKgwsE9h9LcEVXqOJKeGdiuySA27drRtVkeXGltxNtuxucdikAJ77Y7RKVj3cMdYVBt1bw49VzE/s1600/szkop_tdr_nyilt.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEih2u2HLXcTOQ21e49bePmPufDbJZXa91NjkyX91jo9eAEr9VF2wh7MGm3IKJW5b1UwBKgwsE9h9LcEVXqOJKeGdiuySA27drRtVkeXGltxNtuxucdikAJ77Y7RKVj3cMdYVBt1bw49VzE/s640/szkop_tdr_nyilt.jpg" width="640" /></a></div>
<br />
...and with proper termination:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ4bLyRQEsHw3WXgtYp9z0kDsgUTZujEOVNGLXclwh67ySuw4O_9IHNbBXp2rrfj8zlTFzo5roy0d89zoF_AqR6xcpk4zcRNlg2x3EAZrUYt2IqqCPvuf2ECBmOcWpGYH40zkPsg8IMiU/s1600/szkop_tdr_zart.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ4bLyRQEsHw3WXgtYp9z0kDsgUTZujEOVNGLXclwh67ySuw4O_9IHNbBXp2rrfj8zlTFzo5roy0d89zoF_AqR6xcpk4zcRNlg2x3EAZrUYt2IqqCPvuf2ECBmOcWpGYH40zkPsg8IMiU/s640/szkop_tdr_zart.JPG" width="640" /></a></div>
<br />
<h2>
<b>The high voltage</b></h2>
I built the high voltage generator myself. The first version was a simple Zener diode shunted uncontrolled boost converter. This one pulled about 25mA out of a little PP3 9V battery. Googling "9V battery discharge curve" and looking at the first few hits hints at about 20 hours of service time.<b></b><br />
<br />
The current could be reduced to 13mA by redesigning the circuit. This is almost the half of the original current draw, and the service life is likely more than doubled. The new version is actively controlled: one the output voltage is sufficient to open up a chain of Zener diodes, the duty cycle of the PWM signal driving the booster is reduced.<br />
<br />
Here's the final version of the circuit:<b></b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0fVnJJeIkAUXOKZp9uvZiCR2eqmrgqDoaX61ICS3Ri5NAOQ6ZaTuuXvRruXSUjwsgcJYEOXH1TFVssLVj-O-CGQWF3NrWyOrlFw7rK3i264IoaXZNoX3XF0n_TxjJRJLcBv88ka1W_IE/s1600/hv-power-supply-200v.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="1600" height="254" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0fVnJJeIkAUXOKZp9uvZiCR2eqmrgqDoaX61ICS3Ri5NAOQ6ZaTuuXvRruXSUjwsgcJYEOXH1TFVssLVj-O-CGQWF3NrWyOrlFw7rK3i264IoaXZNoX3XF0n_TxjJRJLcBv88ka1W_IE/s640/hv-power-supply-200v.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
This was captured and simulated in LTSpice.<br />
<br />
The LTSpice files for this circuit can be found here: <a href="https://github.com/netom/high-voltage">https://github.com/netom/high-voltage</a><br />
<br />
The circuit charges C4, a 100nF capacitor. Make sure to use one that is rated more than 200V, I used a 630V model just to be safe.<br />
<b></b><br />
The high voltage is generated by Q3 and L1. Q3 is driven by a square wave of varying duty cycle between 50% and 0%. The mosfet opens and closes quickly at about 7Khz. L1, the 10mH inductor kicks up the voltage on the drain of Q3 when it closes. The energy of the kick is determined by how long Q3 has been held open, and that is in turn controlled by the duty cycle of the square wave on it's gate.<br />
<br />
The driving signal is provided by the 555 integrated circuit. It is wired as an astable multivibrator. The ICs CV pin (control voltage) is modulated by Q2.<br />
<br />
Q2 is opened if the voltage on the output (across C4) reaches a voltage where the Zener diode chain - consisting of D3, D4, and D5 - and the BE junction of Q2 is opened. Current flows through R6, D5, D4, D3 and the BE junction of Q2 to the ground. The transistor starts to conduct, and reduces the on the CV pin. (Inside the 555 the CV pin is connected to a simple resistive voltage divider.)<br />
<br />
With the voltage on the CV pin dropping, the 555 reduces the duty cycle of the square wave. Q3 conducts for a shorter time, L1 will provide less of a kick, delivers less charge to C4, reducing the voltage.<br />
<br />
When the output is not loaded, this control loop keeps CV fairly low, close to the saturation voltage of Q2, about 0.1V. Q2 will pull about 1.8mA out of the CV pin to achieve this. Assuming an "average" 2N3904 transistor, the base current will be about 12uA. R6 will help limit the current through the diodes and the base of Q2, and provides low-pass filtering together with C3.<br />
<br class="Apple-interchange-newline" />Q1 provides feedback by liting up or closing off D1. The LED is labelled as "CTRL" on the box, it means "attention, ConTRLol voltage is too high". I choose a random small red LED, the part number on the schematic is just on other random one from LTSpice's library.<br />
<br />
This LED turns on whenever the control voltage is large enough to open Q1 thorugh the voltage divider formed by R4 and R5. D1 is therefore OFF, if the output voltage can be maintained with a low duty cycle PWM, and therefore low current. Presenting a large enough load to the box will cause this LED to turn on.<br />
<br />
If there is significant current draw, D2 lights up. This makes this power supply an excellent cable tester: connecting an unterminated cable to the output should NOT cause D2 to turn on. A shining D2 means that there's significant current draw from the output, therefore the cable is either shorted, or it's leaking a bit at high voltages (water or dirt in the connector, half-broken shielding, etc).<br />
<br />
It is important to place C3 "after" the diodes, to Q2's base. Placing it next to R6 will cause loss of precise voltage control. The screenshot below was recorded on an oscilloscope measuring the voltage between the ground and the conductor between D3 and D4. (Measuring the voltage of the conductor between D4 and D5 yields similar results.) Notice the sawtooth-ish waveform:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFWRgHbd-HEtksARpgMyUP3329L60b8WqjmxkNe1UzeMf-8cSRDhK7IOzrjL9bLBHLFBQysA5EYqYO7UhiGuPorrrZXw2gDJcvVMq4E61UoOXKUI8MDNs8dBsmOKSo-cyelgxRudNFjRs/s1600/pic_8_1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="800" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFWRgHbd-HEtksARpgMyUP3329L60b8WqjmxkNe1UzeMf-8cSRDhK7IOzrjL9bLBHLFBQysA5EYqYO7UhiGuPorrrZXw2gDJcvVMq4E61UoOXKUI8MDNs8dBsmOKSo-cyelgxRudNFjRs/s640/pic_8_1.png" width="640" /></a></div>
<br />
<br />
The jumps on the waveform are cause by D4 or D5 going into conduction quickly, draining charge from C3. Since high voltage Zener diodes conduct in avalanche mode, they might have a negative resistance region. This is seems to be a less-known fact, I found little information on the subject. (Here is some: <a href="https://www.semanticscholar.org/paper/Negative-resistance-characteristics-of-Zener-diode-Zhang-Takaoka/3a041fb37b8744138edda1c9711b9b0d2ca096e2">https://www.semanticscholar.org/paper/Negative-resistance-characteristics-of-Zener-diode-Zhang-Takaoka/3a041fb37b8744138edda1c9711b9b0d2ca096e2</a>)<br />
<br />
So when the diodes just start to conduct, they migh go into negative differential resistance mode, and deliver a quick burst of charge. These bursts are "kicking" on Q2's base, opening the device. The CV pin is yanked low.<br />
<br />
Following the kick, the voltage on C3 drops, and conduction stops briefly. The CV pin is allowed to recover. This results in a jolt of power to be delivered onto the output, causing the CTRL LED to turn on frequently, and shine dimly. The output voltage is noisy, and the overall current drain is higher than it should be.<br />
<br />
Placing C3 right to the base of Q2 means at one hand less charge for the avalanching Zener diodes to work with, and provides some buffer at the base of Q2 where the avalanche noise arrives. The right placement of C3 therefore reduces the avalanche noise by not being upstream so not providing power to the avalanches, and second by being at the base of Q2 and providing a reservoir of capacitance where the (much smaller) avalanches of charge may arrive without causing much disturbance.<br />
<br />
The output of the device is a BNC socket. It is connected across C4, the hot center conductor is connected via a resistor. This can be as large as 100K if the application allows it. A bleed resistor across C4 is also helpful, even if it as large as 100MOhms. <b>I do recommend using a 100K output resistor and a 10-100MOhm bleed resistor across C4.</b><br />
<br />
The device was built in a box made out of single-sided copper clad FR4 sheets. Note that the hot part of the socket is hard to touch, but not impossible to do so:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiBNpjVFXgWh_xkn4bfAFOBaTM5nyZG8D920f5CdKqORubgDk70IDCdIWd-EslKXsFyrQFYe7PO2p1yNpQ0fHt95KUz8cWudZsndGYfzKddgAMtAx15C9pGsN8Yt8F2NxweU3I-32FQ7A/s1600/DSC_0518.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiBNpjVFXgWh_xkn4bfAFOBaTM5nyZG8D920f5CdKqORubgDk70IDCdIWd-EslKXsFyrQFYe7PO2p1yNpQ0fHt95KUz8cWudZsndGYfzKddgAMtAx15C9pGsN8Yt8F2NxweU3I-32FQ7A/s640/DSC_0518.JPG" width="640" /></a></div>
<br />
The ciruit itself was built on a piece of unetched PCB. One side is ground, the other side is +9V. Most componets were soldered onto / over the ground side. This is the first version with simple shunt regulation and discrete transistor astable:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcMkYRWAXfkzGbjO63q0m7MdqpIIC8zSpI_mylK_39TDc2AVzJR1H9vmPXIU3bn2rK9SDjnCAYXoyNNiYldFPryegRnztTKxlYNFv03moEVb7f9YZ6DkrpuUARbjwiyncW9L9M2cwNTtY/s1600/DSC_0521.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcMkYRWAXfkzGbjO63q0m7MdqpIIC8zSpI_mylK_39TDc2AVzJR1H9vmPXIU3bn2rK9SDjnCAYXoyNNiYldFPryegRnztTKxlYNFv03moEVb7f9YZ6DkrpuUARbjwiyncW9L9M2cwNTtY/s640/DSC_0521.JPG" width="640" /></a></div>
<br />
Yes, I did touch hot wires accidentally. Yes, it hurt a lot. It's no fun, don't do it.<br />
<h2>
<b>The pulse generator</b></h2>
Please excuse the hand-drawn schematic:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzfMESEiJiz7VD_jS-hFRyXy3S7AfrFhUKoLWchs4tAdcD43h4XV0xkWgY1TrTWvw4vPTvOmzcd7ZeSJ0V2Qf2uHYZ-peNmEEHXtsqtyx9waTncctvCoHk4_enjF0LtAjCRCWNKpGLdCo/s1600/DSC_0545.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="736" data-original-width="1600" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzfMESEiJiz7VD_jS-hFRyXy3S7AfrFhUKoLWchs4tAdcD43h4XV0xkWgY1TrTWvw4vPTvOmzcd7ZeSJ0V2Qf2uHYZ-peNmEEHXtsqtyx9waTncctvCoHk4_enjF0LtAjCRCWNKpGLdCo/s640/DSC_0545.JPG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<br />
I didn't bother to draw it in LTSpice, because LTSpice doesn't simulate avalanche effects.<br />
<br />
I'm not going to go into details regarding the operation of this circuit. I just really would like to give an overview and some building instructions so it can easily be reproduced. If you'd like to know <i>everything</i>, take a look at this great article by Kerry D. Wong explaining it all:<br />
<br />
<a href="http://www.kerrywong.com/2013/05/18/avalanche-pulse-generator-build-using-2n3904/">http://www.kerrywong.com/2013/05/18/avalanche-pulse-generator-build-using-2n3904/</a><br />
<br />
For an even more detailed description he directs us to this paper:<br />
<br />
<a href="https://icecube.wisc.edu/~kitamura/NK/Flasher_Board/Useful/research/RSI02253.pdf">https://icecube.wisc.edu/~kitamura/NK/Flasher_Board/Useful/research/RSI02253.pdf</a><br />
<br />
For the avalanche transistor we're going to use a simple 2N3904. Other transistors will probably work unless they have a CE breakdown voltage in excess of 150-200V. The collector gets the 200V voltage via a 1MOhm resistor. A small capacitor of 10pF is also connected to the collector and the other terminal of the capacitor goes to ground.<br />
<br />
Although the 2N3904 is rated for 40 volts, most pieces will break down near or even well over a hundred volts. Don't be surprised of the first one out of your drawer can withstand more than a hundred and fifty!<br />
<br />
The capacitor should be one with very low inductance and low ESR. Small, high-frequency NP0 types are probably fine. I used a silver mica capacitor because I had some laying around.<br />
<br />
The base of the transistor should be connected to ground via a 22K resistor. The resistor value was determined by trial and error. This value seemed to achieve the narrowest and tallest pulses (beside of just working at all, of course) out of a handful of other values.<br />
<br />
The circuit is working with frequencies in the multiple hundred megahertz range, so keep every connection as short as possible and/or use transmission lines of proper characteristic impedance and terminations.<br />
<br />
The final device has three terminals:<br />
<ul>
<li>One for the high voltage power supply</li>
<li>One for the pulse output to the measured feedline</li>
<li>One for the oscilloscope</li>
</ul>
I choose to build this circuit into a small, shielded box with three BNC sockets. The "pulse out" sockets' hot pins touch at the middle. The transistor is soldered there with as short a lead as possible. The capacitor and R2 are also soldered with short leads. The lead lengths of R1 are of little significance.<br />
<br />
Pack everything real tight to avoid reflections that smear and blunt the nice and sharp peak.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5lXUAjrs6vYhM3MhN4SI83FqHaVjl3yJGRbS9RKxO1-NK4j7HOjmUdaIveudqdS9Gs-N1Js6i1HlnH2mTDsgIvrbctm77gMeowjy6YZZz0uTHOF8itGi5TWBU4vbk8YlkuNgUwdd4sgI/s1600/DSC_0520.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5lXUAjrs6vYhM3MhN4SI83FqHaVjl3yJGRbS9RKxO1-NK4j7HOjmUdaIveudqdS9Gs-N1Js6i1HlnH2mTDsgIvrbctm77gMeowjy6YZZz0uTHOF8itGi5TWBU4vbk8YlkuNgUwdd4sgI/s640/DSC_0520.JPG" width="640" /></a></div>
<br />
The generator itself is also built into a box made out of single-sided unetched PCB sheets.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9KEdfg2cSldRPBX2qWop_8fHQI1tB2zvF4uQAY6T2Ph2E7hT-eu8HIFg1gCl5Xmw_951F8mEzK809oneZKFNjemGU-9x4PbN1LRbuEUHHgJ8NV5lbfpImzJU_jn72cPz_R4D-gPWqSjw/s1600/DSC_0519.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9KEdfg2cSldRPBX2qWop_8fHQI1tB2zvF4uQAY6T2Ph2E7hT-eu8HIFg1gCl5Xmw_951F8mEzK809oneZKFNjemGU-9x4PbN1LRbuEUHHgJ8NV5lbfpImzJU_jn72cPz_R4D-gPWqSjw/s640/DSC_0519.JPG" width="640" /></a></div>
<br />
The circuit works like this:<br />
<br />
C1 is charged through R1 with about 200 volts, and therefore - at the beginning - 200 microamperes. The avalanche breakdown occures at a random voltage somewhat (even substantially!) higher than what the part is rated for.<br />
<br />
This throws the transistor wide open, and it dumps the charge in C1 onto the 50 ohm feedline. Since the line is 50 ohm in both directions, the transistor will see a 25 ohm resistance (and some inductance because of the leads and wires of and inside the package). C1 empties quickly, the current stops, the transistor closes. C1 starts to charge again.<br />
<br />
When the transistor is closed, it presents high impedance to the feedline, causing virtually no reflections. The signal can "pass under the transistor" unchanged.<br />
<br />
The pulses occur in a somewhat random fashion around a characteristic frequency. This frequency is influenced by R1, C1, the input voltage, and the <i>actual</i> breakdown voltage of the transistor.<br />
<br />
The time between the pulses can be calculated like this:<br />
<br />
t_fire = - log(1 - V_breakdown / V_input) * R1 * C1<br />
<br />
With my particular 2N3904 with 160V (!) actual breakdown voltage, 200V supply, 1Mohm resistor, 10pF capacitor:<br />
<br />
- ln( (1 - 160 / 200 ) * 1 * 10^6 * 10 * 10^-12 = 16.1 * 10^-6<br />
<div>
<br /></div>
<div>
or 16.1 milliseconds, wich means about a 62 kilohertz pulse train.<br />
<br />
The frequency of the output can be changed by changing R1. Larger R1 means lower frequency.</div>
<div>
<h2>
Measurement setup</h2>
<div>
To measure the cable length <i>or</i> valocity factor, the setup should be similat to this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiILEOzD-wJ5yVi3F8lLyB8wXbt3kefrf9AVfX41owf1FI6_Fk5iKaKFGY318AfaG5WnEI1QwbizQMjDisLq_pf6RVM-hLnj0l7-6F-jpY8oI_xAIZqy1aofQR38eUZVmtKx1_I4o4ti_Y/s1600/DSC_0525.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiILEOzD-wJ5yVi3F8lLyB8wXbt3kefrf9AVfX41owf1FI6_Fk5iKaKFGY318AfaG5WnEI1QwbizQMjDisLq_pf6RVM-hLnj0l7-6F-jpY8oI_xAIZqy1aofQR38eUZVmtKx1_I4o4ti_Y/s640/DSC_0525.JPG" width="640" /></a></div>
<div>
<br /></div>
<div>
The generator is powered by the HV power supply via a short pigtail. Don't use a long one, since these cables act as capacitors, and might shock you when charged up.</div>
<div>
<br /></div>
<div>
The two "pulse out" sockets are connected to the cable under test and the oscilloscope. It doesn't matter which one is which.</div>
<div>
<br /></div>
<div>
My oscilloscope only has a high impedance input, so I needed to terminate the cable with a BNC tee adapter and a 50 ohm terminator.</div>
<div>
<br /></div>
<div>
The first pulse is coming straigh from the generator. The second one is being reflected off the unterminated end of the cable.</div>
<div>
<br /></div>
<div>
Look what happens if we terminate the cable properly:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0Myum_tD2RNvx9mqkkSOAl6ZdgeAHQUg5t8KVM0dd_AJijDSWonVUqfNlnLrbutpEYzsZdAQh8DVyvrOFhl4lCmoyT7UOpurOi5Oy9tQFeRZ0XyyPQnz3tqR8vdwJKc37TSTJj259mfA/s1600/DSC_0526.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0Myum_tD2RNvx9mqkkSOAl6ZdgeAHQUg5t8KVM0dd_AJijDSWonVUqfNlnLrbutpEYzsZdAQh8DVyvrOFhl4lCmoyT7UOpurOi5Oy9tQFeRZ0XyyPQnz3tqR8vdwJKc37TSTJj259mfA/s640/DSC_0526.JPG" width="640" /></a></div>
<div>
<br /></div>
<div>
Shorting the end of the cable causes the reflected pulse to be inverted:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsci4dJOM3OFv5NPteDusejEoHVCF_sPC7xo0cmyW3WLkhKKL45vz3NVCne9bxdKSHi0kXA3NYBrEbhGanvLisn6iGmtsRHPefCzm4HBVTnWJXkzrUgOywxK38KlA1bAIvMwDDxI1Z8W8/s1600/DSC_0527.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsci4dJOM3OFv5NPteDusejEoHVCF_sPC7xo0cmyW3WLkhKKL45vz3NVCne9bxdKSHi0kXA3NYBrEbhGanvLisn6iGmtsRHPefCzm4HBVTnWJXkzrUgOywxK38KlA1bAIvMwDDxI1Z8W8/s640/DSC_0527.JPG" width="640" /></a></div>
<div>
<br /></div>
<div>
I had a few ElectroBoom moments with this experiment. I can't whole heartedly recommend this exact experiment setup. If you don't know why, you shouldn't build this. :)</div>
<h2>
Measuring cable length</h2>
<div>
A piece of RG-58 has a velocity factor of 0.66. The question is, how long is it, if the scope screen looks like this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVAU8_bDsYJwO_a76kZxywkZgPK-2OnE1Cdjjp0pE1ESlMFptwxhMCW84n5AjodzenhuncShPHL1x7x2xDZSKYBiwMN4vkaLVwzPyY7X46xOiCYa4Y8BiiZbeBaxj1RjFhrjXm2HsP4F4/s1600/pic_7_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="800" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVAU8_bDsYJwO_a76kZxywkZgPK-2OnE1Cdjjp0pE1ESlMFptwxhMCW84n5AjodzenhuncShPHL1x7x2xDZSKYBiwMN4vkaLVwzPyY7X46xOiCYa4Y8BiiZbeBaxj1RjFhrjXm2HsP4F4/s640/pic_7_2.png" width="640" /></a></div>
<div>
<br /></div>
<div>
The time between the pulses is 164 nanoseconds. Since the velocity factor is 0.66, the speed of signal propagation is 198e6 m/s.</div>
<div>
<br /></div>
<div>
So in 1 second, the signal travels 198*10^6 meters. In 164 nanoseconds, or 164*10^-9 seconds, it travels:</div>
<div>
<br /></div>
<div>
0.66 * 300e6 * 164e-9 / 2 = 16.236 ~= 16 meters and 23 centimeters.</div>
<div>
<br /></div>
<div>
The velocity factor is multiplied by the speed of light to yield the real signal propagation speed. Multiplied by the measured time we get length. Dividing by two is necessary since the signal travels along the cable twice.</div>
<h2>
<b>Measuring velocity factor</b></h2>
The velocity factor can easily be calculated by measuring the cable length and the time it takes for a pulse to travel along it down and up again.<br />
<br />
Using the numbers above, if I have a cable 16 meters and 23 centimeters long, and I measure 164 nanoseconds between pulses, then I can calculate the velocity factor like this:<br />
<br />
( 16.23 / 300e6 ) / ( 164e-9 / 2 ) ~= 0.66<br />
<br />
First the cable length is divided by the speed of light to yield the time it takes for and electromagnetic signal to travel the given length in empty space. This time is divided by half of the measured time (since the signal travels along it twice).<br />
<br /></div>
Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-71765882978899543722019-09-29T02:03:00.001-07:002020-07-21T05:57:53.872-07:00Installing Kubernetes on my Ubuntu laptop<h2>
Why?</h2>
<div>
Since the company I'm working with uses Kubernetes in production, I like to have a sandbox handy to try upgrades and just for general experimentation.</div>
<h2>
Prerequisites</h2>
<div>
I'm using Ubuntu 19.04 "Disco Dingo".<br />
<br />
I'm going to install Kubernetes v1.15.4. The current newest version works a bit differently that makes the latest stable flannel release not to work properly. The procedure below can probably be repeated with now-current (in reader-time :) ) versions as long as they're compatible.<br />
<h3>
Docker</h3>
The docker I choose is version <b>18.09</b>. Check the supported docker version with <i>your</i> Kubernetes version. Do not skip this step. I found this information in the release notes for v1.15: <a href="https://v1-15.docs.kubernetes.io/docs/setup/release/notes/">https://v1-15.docs.kubernetes.io/docs/setup/release/notes/</a><br />
<h3>
Flannel</h3>
</div>
<div>
I'm using flannel with the local installation. The latest stable at this time is v0.11.0. I found no information on the compatiblity of different flannel and Kubernetes versions, so I just tried the latest of both and I got lucky the first time.<br />
<br />
If I'd start the installation today, I'd be out of luck since flannel v0.11.0 does not work with Kubernetes v1.16.0. The flannel manifest on the master branch works fine, but I'd rather use a stable release, and hence I'd go back to use Kubernetes v1.15.4.<br />
<br />
The problem with flannel v0.11.0 and Kubernetes v1.16.0 is that it stopped serving certain deprecated APIs, namely DaemonSet has been moved from extensions/v1beta1 to apps/v1beta2 (see <a href="https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/">https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/</a> for more details).<br />
<br />
Even after fixing the v0.11.0 flannel manifest, the CNI wouldn't work properly, and "kubectl describe node <nodename>" shows "network plugin is not ready: cni config uninitialized". This is probably an other incompatibility between my particular flannel - Kubernetes versions.<br />
<h2>
Network setup</h2>
<div>
I'd like to run the (one-node) cluster on a laptop. This laptop may or may not have network connectivity, and both the public facing interface and IP address will change a lot.</div>
<div>
<br /></div>
<div>
Unfortunately Kubernetes does not like to work with loopback interfaces. I couldn't get Kubernetes </div>
<div>
to use the loopback address neither as the advertised API server address nor as the node IP.</div>
<div>
<br /></div>
<div>
So I needed an interface with a static IP regardless of Internet connectivity.</div>
<div>
<br /></div>
<div>
Enter the dummy interface.</div>
<div>
<br /></div>
<div>
A dummy interface on Linux is really just another loopback device: it does the same thing really without <i>actually being</i> a loopback device. This is important, since I could find lines in the Kubernetes code that were meant to actively avoid using the loopback device. Since a "dummy" is not "loopback", Kubernetes will be happy with one, and will need little persuasion to use this dummy interface with a static address.</div>
<div>
<br /></div>
<div>
So let's edit /etc/network/interfaces and add these lines:<br />
<br />
auto aaa0<br />
iface aaa0 inet static<br />
<span style="white-space: pre;"> </span>address 172.31.0.1/16<br />
<span style="white-space: pre;"> </span>pre-up modprobe dummy; ip link add aaa0 type dummy<br />
<span style="white-space: pre;"> </span>post-down ip link del aaa0<br />
<br />
I'm using a name that will make the interface appear on the top of the list provided by "ifconfig". I don't know if this has any importance in our case. You could probably use any name.</div>
<div>
<h2>
Docker setup</h2>
</div>
</div>
<div>
Let's install the appropriate docker-ce version, but first let's make sure no conflicting version is installed:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt remove docker docker-engine docker.io</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt install apt-transport-https ca-certificates curl software-properties-common</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt-add-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu cosmic stable"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
You can view the available version with apt-cache madison:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ apt-cache madison docker-ce</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:19.03.2~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:19.03.1~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:19.03.0~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.9~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.8~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.7~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.6~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.5~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.4~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.3~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.2~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">docker-ce | 5:18.09.1~3-0~ubuntu-cosmic | https://download.docker.com/linux/ubuntu cosmic/stable amd64 Packages</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
I'd like to use version 18.09, so I'll issue the following commands:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt install docker-ce=5:18.09.9~3-0~ubuntu-cosmic</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt-mark hold docker-ce</span></div>
<div>
<br />
Let's create the file /etc/docker/daemon.json with the following content:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "exec-opts": ["native.cgroupdriver=systemd"],</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "log-driver": "json-file",</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "log-opts": {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "max-size": "10m",</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "max-file": "10"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "storage-driver": "overlay2"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<br />
The most important part is the cgroup driver set to systemd. I'm going to use overlay2 as the storage driver merely because it works and does not require any further setup. Logging will take place in the JSON format, and log-opts will prevent docker from keeping too much of them.<br />
<br />
Since this is my laptop we're talking about here, I don't like docker or Kubernetes to start automatically, I don't need them all the time, especially not when I'm using the laptop while travelling or otherwise being contrained by the battery, so I'll go ahead and prevent docker from starting automatically, and also restart it for the above configuration to take effect:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ systemctl disable docker</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ service docker restart</span><br />
<br />
Docker is ready and should be running (check it with "service docker status").<br />
<h2>
Installing Kubernetes</h2>
Let's add the appropriate public key to apt's keystore, add the repository, and install the correct version. (Hint: use apt-cache madison to check for available versions).<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt install kubelet=1.15.4-00 kubeadm=1.15.4-00 kubectl=1.15.4-00</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo apt-mark hold kubelet kubeadm kubectl</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo systemctl disable kubelet</span><br />
<br />
That's it, we have all the software installed on the local system for Kubernetes to work.<br />
<h2>
Setting up Kubernetes</h2>
I'm going to use kubeadm to take care most installations steps. Let's write out the default configuration first:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ kubeadm config print init-defaults | sudo tee /etc/kubeadm.conf</span></div>
<div>
<br />
Open that file in an editor, and look for the "networking" key. Add the "podSubnet" under it.<br />
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">networking:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ...</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> podSubnet: 10.244.0.0/16</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ...</span></div>
<div>
<br />
At the end of the file, add the following snippet:<br />
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">---</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">apiVersion: kubelet.config.k8s.io/v1beta1</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">kind: KubeletConfiguration</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">failSwapOn: false</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">evictionHard:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> imagefs.available: 5%</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> memory.available: 100Mi</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> nodefs.available: 5%</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> nodefs.inodesFree: 5%</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
This will make sure that Kubelet will start with swap on.<br />
<br />
The other thing I'm going to deal with is the node IP. Kubelet will still going to use the public-facing interface (eth0 or wlan0 in my case) and the IP address of it as the node address. Kubelet's documentation mentions the --node-ip option.<br />
<br />
Reading kubelet's systemd service unit file revealed that extra options can be passed to kubelet via the <b>KUBELET_EXTRA_ARGS</b> variable. The variable should be defined in <b>/etc/default/kubelet</b>:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">KUBELET_EXTRA_ARGS="--node-ip=172.31.0.1"</span></div>
<br />
We can now fire up kubeadm and it will set up our single node cluster.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ sudo kubeadm init --config=/etc/kubeadm.conf --ignore-preflight-errors=Swap</span><br />
<br />
I need to use the --ignore-preflight-errors=Swap option to tell kubeadm that it is OK to proceed with swap enabled. Kubeadm and Kubelet are two independent components, each of them must be told to ignore swap being turned on.<br />
<br />
I'm going to get warnings about the swap, and that docker is disabled, but that's OK.<br />
<br />
To be able to use kubectl I need authentication. A configuration file with the appropriate credentials is written to /etc/kubernetes/admin.conf. Copying it to ~/.kube/config enables me to use kubectl:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ mkdir $HOME/.kube</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ cp /etc/kubernetes/admin.conf $HOME/.kube/config</span><br />
<br />
The cluster might need a minute or two to start up, but after that, I should see some activity:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ kubectl get -A pods -o wide</span><br />
<br />
There sould be several pods running, and coredns pending.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ kubectl get nodes</span><br />
<br />
reveals that the node is not ready yet. The problem is that I didn't install any CNI yet. So go ahead and let's install flannel:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml</span><br />
<br />
The flannel manifest - among other things - contains DaemonSet definitions for kube-flannel-ds. This pod should start up shortly, and coredns should follow in a minute or so.<br />
<br />
The cluster is ready for use. Just let's not forget to start it manually when needed:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ service docker start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ service kubelet start</span><br />
<br />
The cluster need a minute or two to start, be patient. When not needed, the cluster can be stopped:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ service kubelet stop</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ service docker stop</span><br />
<br />
The last command will stop any running containers, so do use it to completely shut the cluster down.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-4462078193459486622019-04-11T03:34:00.003-07:002023-11-16T21:31:41.806-08:00Building a radiation detector from scratch 4 - Detector MK IISince I'd like to keep up with the head-spinning pace of interesting articles, let me present you this year, :) the<br />
<br />
In the last episode I described an ion chamber radiation detector. It works very well: stable and sensitive enough for doing simple experiments.<br />
<br />
It can be improved though in several ways. Here follows a series of ideas and a circuit diagram. :) I did not build this yet, but I'll build it "soon". In this decade, or the next. :P<br />
<br />
<a name='more'></a><b>High voltage</b><br />
<b><br /></b>
The voltage potential of the outer shell of the detector (the tin can) could be raised up to several hundred volts to provide more sensitivity.<br />
<br />
Teralab did wonderful experiments: <a href="http://www.teralab.co.uk/Experiments/Ion_Chamber/Ion_Chamber_Page1.htm">http://www.teralab.co.uk/Experiments/Ion_Chamber/Ion_Chamber_Page1.htm</a>.<br />
<br />
According to their results, it's well worth increasing the voltage between the transistor's base and the outer shell up to 200V, after that we get diminishing returns.<br />
<br />
Raising the voltage could be done in several ways in practice. One could by a DC-DC converter module used to charge flash capactiors, or build a (rather large) charge pump, homebrew a simple boost converter, drive a PCB mains transformer in reverse, etc. Just make sure the voltage source is stable enough at least in the short term. Any noise from the supply will be picked up by the <i>very</i> sensitive detector circuit.<br />
<b><br /></b>
<b>Isolation and safety</b><br />
<b><br /></b>
Working with voltages in excess of a few dozen volts poses a shock hazard. Careful measures should be taken to minimize the danger of shock during work and use.<br />
<br />
It's best to put everyting into a solid metal housing, and don't let <i>any</i> leads or pins out that has dangerous voltages on it.<br />
<div>
<br /></div>
<div>
Because of these, I decided to use optocouplers as the output on a separate battery, and also to separate the high voltage power supply. This makes it possible to build a modular solution, whereby you can choose a separate high voltage (or not so high voltage) power supply that is designed and built separately.</div>
<div>
<br />
One can use the internal battery also simply by shorting out the high voltage power supply connector. Since under no condition should any significant current flow through that connector, a simple 50 Ohm off-the-shelf termination resistance can be used also.</div>
<div>
<br /></div>
<div>
Below is the schematic of the Mark II Ion Chamber Radiation Detector:</div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyY1X1uZg3nQxrxYHRqbc7v8-cp7Y6O85g6LRGO9nODqRrO81zhLND4VsCj4EeFamY0WCA3Zt3aGyRK3yN0wWgKWaCvLmmq4TG7kBiJ81brdC6mM7ZGWsDKzsNHU69jaRs6iXDMPNgGDk/s1600/mk2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="603" data-original-width="1600" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyY1X1uZg3nQxrxYHRqbc7v8-cp7Y6O85g6LRGO9nODqRrO81zhLND4VsCj4EeFamY0WCA3Zt3aGyRK3yN0wWgKWaCvLmmq4TG7kBiJ81brdC6mM7ZGWsDKzsNHU69jaRs6iXDMPNgGDk/s400/mk2.png" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<b>Single-ended output</b><br />
<br />
A simple operational amplifier circuit might be used to deliver a single-ended voltage output. The OPA could use it's own power suply, or utilize the optocoupler's battery. Therefore it is recommended to create three output pins / sockets / leads: "output" (left resistor top lead), "reference" (right resistor top lead), and "Vcc" (optocoupler Vcc).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhN2FOg9iLKn8hbm2OWGAEiRsbh9eYAXRzQlon0qC4rOvJIZ8df-1nFFOrHnHGHYK5lR_sbW6l8AXJmDTg3Syhrr1q_zohDReEsHZ4bwtzceFxcrWrggMXCVjgnqC6QTjjNgnTRdQluyjk/s1600/opa-single-ended-output.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="841" data-original-width="1162" height="288" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhN2FOg9iLKn8hbm2OWGAEiRsbh9eYAXRzQlon0qC4rOvJIZ8df-1nFFOrHnHGHYK5lR_sbW6l8AXJmDTg3Syhrr1q_zohDReEsHZ4bwtzceFxcrWrggMXCVjgnqC6QTjjNgnTRdQluyjk/s400/opa-single-ended-output.png" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
Resistors should be choosen so the voltages respect the OPA's capabilities. The "SingleOut" output will be our single-ended output. Since this circuit can only output positive voltages, "SingleOut" will peg to zero if "Output" goes below "Reference".<br />
<br />
This should't happen in theory, since there always will be a slight background radiation, and therefore some current in the chamber. If the circuit starts to drift however, it might happen, and ruin measurements in case of an unattended setup. Since the last version of the chamber proved to be surprisingly stable, I believe this will hardly be a problem.<br />
<br />
<b>Data logging</b></div>
<div>
<b><br /></b>There are a lot of possibilities for recording and/or transmitting voltage data. The single-ended output is easily interfaced with and arbitrary device capable of measuring and displaying / logging / transmitting voltages. This can be an analog uamp meter with a suitable resistor in series, a digital voltage meter module, a DVM, and arduino with an SD-card or Bluetooth module, etc.<br />
<div>
<br /></div>
<div>
I'm going to use the MK II with a DVM and probably try to hook it up to an Arduino Nano, to interface it with a computer throug USB and log data, and might even set up the Arduino to log the data to an SD card.</div>
<br />
Next time we'll take a look at the built detector and the measurement results.</div>
Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-41471517904322357212018-01-26T07:07:00.003-08:002018-01-26T07:07:57.092-08:00Contacting the International Space StationThe International Space Station (ISS) is a huge (>400 tonnes mass, >900m3 pressurized volume) habitable satellite on low earth orbit. It orbits the Earth at about 400km high with a speed of 7.67km/s (27,600 km/h), and goes around it in about every 93 minutes.<br />
<br />
It's permanently crewed, can support up to 6 astronauts, and provides a platform for many scientific experiments. Besides astrobiology, astronomy, materials science, and many other fields, it hosts experiments for the ARISS - "Amateur Radio on the International Space Station" project.<br />
<br />
ARISS experiments give opportunities to observe or even contact the station with simple amateur radio equipment.<br />
<br />
<a name='more'></a><b>Activities and Equipment</b><br />
<br />
There are various activities onboard that can be monitored, or taken part of with relatively simple equipment. No large parabolic antenna and automatic tracking is needed.<br />
<br />
<b></b>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg51MMU_hkKNoITGFkmIm45hNLNGuhvPi8Uru717kOQDXAJmmQ_rl8pGs71Rp8gVzjTdlW4CrAZOpyCihyphenhyphenUVf9VtBnNsoA0DjdMTeoklj2RkuOXqZrLgilumxG-701aXAgojpnpQXUEa1c/s1600/iss-range-calculation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="360" data-original-width="597" height="192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg51MMU_hkKNoITGFkmIm45hNLNGuhvPi8Uru717kOQDXAJmmQ_rl8pGs71Rp8gVzjTdlW4CrAZOpyCihyphenhyphenUVf9VtBnNsoA0DjdMTeoklj2RkuOXqZrLgilumxG-701aXAgojpnpQXUEa1c/s320/iss-range-calculation.png" width="320" /></a></div>
<br />
<br />
At the time of writing this post, the ISS is 403 Km high. The figure above shows how it's distance can be calculated when it just appears on the horizon. At that time, it is merely 2300 Km away. This is the maximum distance we need to cover.<br />
<br />
The calculation goes like this: D is where we stand, E is the ISS, C is a point directly under it, A is the centre of the Earth. Earth's radius is about 6371 kilometres. The AD radius is perpendicular to the plane tangent to the Earth's surface at point D. E is being on the horizon, ED is perpendicular to AD, so AED is a right angled triangle, the distance we need to calculate is ED. AD is known, AC and EC, therefore AE is also known. Pythagoras will do the rest.<br />
<br />
In practice, you only want to try and contact / listen to the station when it is at least 10-15° up, and it is somewhat closer.<br />
<br />
That does sound like a great distance. However anyone who participates in amateur radio knows that such a distance can easily be covered with a couple, or couple of dozen watts of power and cheap antennas even on the shortwave bands and with skywave propagation.<br />
<br />
Since the ISS is high above, there's nothing between us to shield the signals. The ARISS experiments are usually working on the 2 meters (145 MHz) and 70cm (433 MHz) bands. It is quite practical to build or buy good directional antennas for these bands. A Yagi-style one with a reflector and two directors yields excellent results on 2 meters, a few more on 70 centimetres is probably a good idea.<br />
<br />
The mode of operation on these bands is almost always FM, so a simple handheld transceiver (or scanner) will do the job just fine. I use my Kenwood TH-D72E. This is a more pricey HT, and it's a very good one, but a cheap one would also do OK.<br />
<br />
All in all, what you really need is a handheld radio or scanner, a portable (not too large) Yagi antenna for the band, and you're good to go.<br />
<br />
...but what can you do with it?<b><b> </b></b><br />
<br />
<b><b>Contact with ground crew</b></b><br />
<br />
Astronauts sometimes use simple FM radio to keep contact with the ground crew in Russia on 143.625<b><b></b></b>MHz.<br />
<br />
An example of receiving these signals can be seen here (and may videos on YouTube): <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/F5gByVuTpmQ/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/F5gByVuTpmQ?feature=player_embedded" width="320"></iframe></div>
<br />
<b><b>School contacts</b></b><br />
<br />
The astronauts onboard often participate in events called "school contacts". During such an event a radio link is established between a school and the ISS. If the contact is voice only, then it is often made at 145.800 MHz, and can easily be listened to with the setup described above.<br />
<br />
<b><b></b></b>
Note that during the event only the ISS can be heard, the school will very likely be out of range - to be more precise - be obstructed by buildings, hills, or the curvature of the earth.<br />
<br />
<b><b></b></b>
An example of receiving the ISS during a school contact at 145.800 MHz:<br />
<br />
<b><b></b></b>
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/mhfQiVqZkts/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/mhfQiVqZkts?feature=player_embedded" width="320"></iframe></div>
<br />
<b><b>SSTV</b></b><br />
<br />
On special occasions the ISS transmits images at 145.800 MHz using FM and SSTV, often in the PD180 mode.<br />
<br />
An SSTV image is just a burst of sound of almost musical quality. To decode it and turn the sound into an image, the radio must be connected to a computer or smart phone, and a decoder program must be running.<br />
<br />
I use a SignaLink USB device (<a href="http://www.tigertronics.com/slusbmain.htm" target="_blank">http://www.tigertronics.com/slusbmain.htm</a>) to connect my handheld Kenwood TH-D72E radio to my Android phone with an USB-OTG adapter. <br />
<br />
I use QSSTV on the PC and an app called Robot36 on my Android phone for SSTV.<br />
<br />
Space Comms on YouTube has a nice video of receiving an SSTV transmission :<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/7to9uX1sWC4/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/7to9uX1sWC4?feature=player_embedded" width="320"></iframe></div>
<br />
Here's the image I managed to capture during the last activity:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSO8d0VCoav4TBIoD32nUm7GAFMuOgNY3SXLojtueN9pXlIphkcuRz6t9QGGGfUes82R6Jtg146E2HTSUGIRgx2scKJ_McKtm4Xv5o-jYlQPtJSdQxRKV0Qv-A5QgRY2meoidJ1fTAYPc/s1600/iss-sstv-1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="496" data-original-width="640" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSO8d0VCoav4TBIoD32nUm7GAFMuOgNY3SXLojtueN9pXlIphkcuRz6t9QGGGfUes82R6Jtg146E2HTSUGIRgx2scKJ_McKtm4Xv5o-jYlQPtJSdQxRKV0Qv-A5QgRY2meoidJ1fTAYPc/s320/iss-sstv-1.jpg" width="320" /></a></div>
<br />
<br />
<b>APRS</b><br />
<b><br /></b>
APRS is a method of transmitting small data packets over radio, most of the time this means 2 meters, and FM.<br />
<br />
<b></b>
APRS nodes may form a network where each node might repeat certain packets, so they jump from node to node and can cover great distances despite each node being a simple, low-power transceiver.<br />
<br />
<b></b>
It is most often used to transmit position and some textual information. This can be observed on <a href="https://aprs.fi/" target="_blank">https://aprs.fi/</a>, a site showing currently active APRS nodes, and lots of information about the network.<br />
<br />
<b></b>
There is an APRS repeater (or digipeater) onboard the ISS, operating at 145.825 MHz simplex (this is both the uplink and downlink frequency).<br />
<br />
<b></b>
Sending APRS packets via the ISS counts as a two-way contact, and you can get a proper QSL card because of that:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0xm6l_2wZwaCJf0DBlclPjEHEEHLC1ZsfWNWEcfx1gfvk2Dz3NDIai76zmUAW57H9AqRa_0uIRSNtH5Z_WZgovP-H2nAsKyxGBGKJ_1_AHgcQ02224P6oTlJdM4diJfPRkJ2fQZhzHdk/s1600/iss-qsl.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1066" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0xm6l_2wZwaCJf0DBlclPjEHEEHLC1ZsfWNWEcfx1gfvk2Dz3NDIai76zmUAW57H9AqRa_0uIRSNtH5Z_WZgovP-H2nAsKyxGBGKJ_1_AHgcQ02224P6oTlJdM4diJfPRkJ2fQZhzHdk/s320/iss-qsl.jpg" width="320" /></a></div>
<br />
<b></b>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFGuSEsj_cMxaNJaqg6LI5UdsdoE9FNTyD2k9rkYZpHZ4ZFfBI_h7sASJdTJdJphk2dPy01NnV3PsP76kIIq4RLIk9DjAfmHL_f5cJPF5yopra5Vjbe3EGSi0zIcnIIhjbGKwSrRv3nEQ/s1600/iss-qsl-back.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1041" data-original-width="1600" height="208" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFGuSEsj_cMxaNJaqg6LI5UdsdoE9FNTyD2k9rkYZpHZ4ZFfBI_h7sASJdTJdJphk2dPy01NnV3PsP76kIIq4RLIk9DjAfmHL_f5cJPF5yopra5Vjbe3EGSi0zIcnIIhjbGKwSrRv3nEQ/s320/iss-qsl-back.jpg" width="320" /></a></div>
<br />
I also made a video of the contact. I used the radio's internal APRS capabilities.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/lbxjc_0_uX4/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/lbxjc_0_uX4?feature=player_embedded" width="320"></iframe></div>
<br />
<b><br /></b>
<b>Two-way voice contacts (proper QSO-s)</b><br />
<br />
Sometimes, rarely, it is possible to make real, two-way contact with someone onboard the ISS.<br />
<br />
Astronauts of course have a LOT of work to be done on the station, and they have very little free time. As far as I know, the last astronaut to operate the HAM station on board was Colonel Doug Wheelock in 2010. KF7ETX David's dedication were rewarded with great luck as he managed to make a QSO with Col. Wheelock. See:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/mSIAi-t0OQI/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/mSIAi-t0OQI?feature=player_embedded" width="320"></iframe></div>
<br />
Usually during such contacts the uplink and downlink frequency is different to prevent interference. In this case uplink was 144.490
MHz and downlink was the usual 145.800 MHz.<br />
<br />
<b></b>
Good luck chasing the 'station and 73!<br />
<b></b>Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-24771547967364976902016-10-27T04:10:00.004-07:002016-10-27T04:13:35.771-07:00Building WSJT-X 1.7.0-rc1 from sources on Ubuntu 16.04.1 XenialWSJT-X is a great desktop application for communicating via radio using various weak signal modes. I use it on the HF bands to contact amateur radio operators literally all over the world.<br />
<br />
The new 1.7.0-rc1 alpha release features a new algorithm for decoding, and it is quite a bit more effective than the previous 1.6.x versions. One of the most important parts is decoding Reed-Solomon codes. The previous versions used a proprietary decoder, this one uses a new, open-source method.<br />
<br />
Since I work a lot in JT65 and JT9 on short wave, building the new version definitely worth the struggle.<br />
<br />
<a name='more'></a>I've set up an LXC container to start fresh. Repeating the commands below will get you a fresh and crispy wsjtx binary (and others), along with the documentation. <br />
<br />
Let's start with creating a new empty directory. I use a directory named "install" to download and build various packages.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mkdir install</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cd install</span><br />
<br />
We'll put everything there. Before we start, let's install packages necessary for building hamlib and wsjt-x. This command will download and install quite a lot of packages. Go grab a coffee.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo apt install man git subversion ca-certificates build-essential autoconf automake libtool texinfo gfortran asciidoctor libfftw3-dev pkg-config qtbase5-dev qtmultimedia5-dev libqt5serialport5-dev asciidoc</span><br />
<br />
Install a recent version of hamlib from github:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ git clone https://github.com/N0NB/hamlib.git</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>cd hamlib</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>git branch for-wsjtx</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>git checkout for-wsjtx</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>git reset --hard f778fe1677bffed68d52f04a440e887f249da56b</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>./autogen.sh</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>./configure</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>make</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>sudo make install</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ cd ..</span></span><br />
<br />
Fetch WSJTX 1.7.0rc1 sources from SourceForge: <br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ svn co svn://svn.code.sf.net/p/wsjt/wsjt/tags/wsjtx-1.7.0-rc1</span><br />
<br />
Create a directory for the release and build the software:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>cd wsjtx-1.7.0-rc1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>mkdir release</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>cd release</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>cmake ..</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">$ </span>sudo cmake --build . --target install</span><br />
<br />
The binaries will be installed into /usr/local/bin. This is normally in the $PATH environment variable, so wsjtx can be launched with:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ wsjtx</span>Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-62202979331576060102016-10-14T07:59:00.001-07:002023-11-16T21:31:29.939-08:00Building a radiation detector from scratch 3 - a stabilized detectorIn this chapter I'm going to show you how to build an atmospheric ion chamber for detecting radiation. The detector will have little drift, can be set to zero, and can be used to compare the radioactivity of samples, measure weak sources, or follow the decay of short half-life sources for hours or days even.<br />
<br />
These features all require that the detector to be stable, and the readings don't drift around with temperature.<br />
<br />
Temperature drift is very likely the worst offender to our previous ion chamber design, so we're going to deal with that one this time.<br />
<br />
<a name='more'></a><b>An improved circuit</b><br />
<br />
The next design will deal with the temperature drift by simply duplicating the amplifier, and use only one for detection. The other one will not be connected to anything, and will only provide the zero-point current, and temperature-dependent error.<br />
<b> </b><br />
We will simply subtract the output of the duplicated circuit from the detector circuit to give us a reading without the baseline current and the temperature error.<br />
<br />
Since no one will be able to build two EXACTLY matching amplifiers, we'll provide a way to cancel any small residual error, and match the reading to zero at background radiation levels.<br />
<br />
In the previous design we've used a DVM's voltage metering function to read really small currents. This can't be done simply enough with two matching amplifiers, so we're going to use 3 transistors per amplifier, a total of <b>6 transistors. </b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXDkaeyM0oZy2kNMiLIyH4hkFPj4IjbquTsF-kt_LMxXcbvFowz7M5AfeoxexnRia3l-aplCZE_dhb1S-awEwZQBl7UcUjqOf2vNi57hhFmrnfBykqdMg5GRxWrIjUo9vZ9gk9HxUE-b8/s1600/ionchamber-circuit.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXDkaeyM0oZy2kNMiLIyH4hkFPj4IjbquTsF-kt_LMxXcbvFowz7M5AfeoxexnRia3l-aplCZE_dhb1S-awEwZQBl7UcUjqOf2vNi57hhFmrnfBykqdMg5GRxWrIjUo9vZ9gk9HxUE-b8/s400/ionchamber-circuit.png" width="400" /></a></div>
<br />
The image above is a snapshot from an electronics simulation application called LTSpice. It's freely available from Linear Technology: <a href="http://www.linear.com/designtools/software/" target="_blank">http://www.linear.com/designtools/software/</a>.<br />
<br />
This application helped a lot in improving the circuit, especially the response to changing input (e.g. fluctuations in radiation). It also doubles as a general schematic capture tool.<br />
<br />
The circuit is basically a Siamese Darlington on steroids. Q1-Q2-Q3 forms a "Darlington triplet". This triplet connects to the ion chamber probe. Q4-Q5-Q6 forms a reference stage.<br />
<br />
Now, there is a catch to Darlington pairs and triplets. Since the hFE of a transistor at very low currents is smaller than at the current it was designed for, simply connecting two transistors won't result in a gain that is exactly the product of the hFE values of the two transistors measured independently.<br />
<br />
A three-stage "Darlington" like this will have <i>much</i> less amplification then the hFE specified in the datasheet on the third. Unfortunately the simulator program I've used - called LTSpice - doesn't seem to account for this effect. I've built the circuit using the simulated results nevertheless, since I had no other source of information at that time.<br />
<br />
I've tried quite a few variations and the above version gave the best results. I've tried placing the resistors between the emitters and the ground, but it gave a very slow (simulated) response. The circuit described here gave reasonably good step response.<br />
<br />
The graph below shows the voltage across a collector resistor while sending a 1pA (picoAmpere) current impulse that is 0.1 seconds long. It can be seen that the circuit is somewhat slow to respond, but considering the ions' flight time in the chamber this actually isn't a problem.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjlLxa0xhNMruhGA4WJ8_SaPwQ5hCKQAlhn08yOwsPl1XwphwTKI_HyHQ11N9k_GBrdtQX6V-uYLRb-qASk8odZsBvMy_ZsrJgakaZnde51UmTWB7CcPdgg-Q_eS74pwav4YJFiMBCo8s/s1600/ionchamber-simulated-response.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjlLxa0xhNMruhGA4WJ8_SaPwQ5hCKQAlhn08yOwsPl1XwphwTKI_HyHQ11N9k_GBrdtQX6V-uYLRb-qASk8odZsBvMy_ZsrJgakaZnde51UmTWB7CcPdgg-Q_eS74pwav4YJFiMBCo8s/s320/ionchamber-simulated-response.png" width="320" /></a></div>
<br />
<br />
Because the transitors don't amplify all that much at picoampere currents, the magnitude of the response is actually an order of magnitude lower. Also because of this effect the voltage drop on the resistors will be much lower, since the currents through them will be less.<br />
<br />
Since this<b> </b>device won't be used to detect single events, a fast response is not absolutely necessary. It would be great to detect single alpha particles from weak sources or single muons from cosmic radiation, and I still believe it's possible given a proper data logger and some kind of display mechanism. Maybe I can conjure up some solution for this in a future article.<br />
<br />
<b>Selecting transistors</b><br />
<br />
Just like in the previous design, choose a bipolar NPN transistor type that has a high "hFE" or "beta" parameter, e.g. it will amplify currents pretty much. Think about 400-800. These types of transistor are cheap and popular. I'll use BC547C-s.<br />
<br />
What is even more important is that we have to build two <b>very similar</b> amplifiers. To achieve this, we must select <b>3 pairs</b> of transistors with very similar hFEs.<br />
<br />
Do not try to match all 6 transistors. It'll take a LOT of work. Do your best and select 3 pairs. Each pair should be as close to each other as possible, but you don't need to match across pairs. Buy 30 or 50 transistors, so you'll have as many options as possible. Use a DVM with a hFE measurement function and measure your transistors. Pair up the similar ones. <b>Do</b> try and select pairs that are <b>very </b>similar.<br />
<br />
Assemble the circuit so the transistors in a pair will occupy the same position in different amplifiers (one in the detector, and one in the dupe).<br />
<br />
Without proper matching the baseline error will be huge, and even after zeroing the reading with a trimmer resistor there will be severe temperature drift.<br />
<br />
<b>Building the detector</b><br />
<br />
The circuit was built on the <i>non-coppered side </i>of a small piece<b> </b>of PC board. The components were fixed with super glue.<br />
<br />
Make sure the glue does not touch any lead as it might conduct slightly, and utterly ruin the readings.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6XPeX897j95hcpwxRXT8XD-ymvI8lVPjy0znh8AvbDpnxxhyphenhyphen1TCH1jN9IxS1I2SkIbrGaAONX6B06rzI-uUh1UGG4UuXPG-3dkELIl6oHlyumVhI-E2a_odVIH1ivyXmPFFcA0_BbK8E/s1600/DSC_0333.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6XPeX897j95hcpwxRXT8XD-ymvI8lVPjy0znh8AvbDpnxxhyphenhyphen1TCH1jN9IxS1I2SkIbrGaAONX6B06rzI-uUh1UGG4UuXPG-3dkELIl6oHlyumVhI-E2a_odVIH1ivyXmPFFcA0_BbK8E/s640/DSC_0333.JPG" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Final assembly, top view</td><td class="tr-caption" style="text-align: center;"><br /></td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
<br />
A battery or power supply can be connected to the tin can (+) and the wire connecting the emitters (-). The DVM should be set to millivolts and connected to the terminals on different resistors where they meet the collectors of their respective transistor (on the bottom leads on this picture). <br />
<br />
<br />
The transistor pairs were glued together so they stay close together, and therefore will exhibit very similar hFE no matter the ambient temperature. Pay attention to the transistor leads, as this arrangement <i>the pairs are not mirrored, but rather rotated</i>. See these photos and try spotting the difference on each side:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4IGLN15uezUURo23TxfhMoFOpkUHN-ltmg2Ip3uzHt9PR1qpd52pZ5PWt6ZrV7WMWB8xFIqj96KufzRks9yUywwRKTEbw_tbe46bzx3NPfftzjHM6gNWLXXCWUyiLQ3Ba5Rs1_HESIe4/s1600/DSC_0334.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4IGLN15uezUURo23TxfhMoFOpkUHN-ltmg2Ip3uzHt9PR1qpd52pZ5PWt6ZrV7WMWB8xFIqj96KufzRks9yUywwRKTEbw_tbe46bzx3NPfftzjHM6gNWLXXCWUyiLQ3Ba5Rs1_HESIe4/s640/DSC_0334.JPG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Final assembly, right side</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrBcWmIKsNm70KhVpzhbI7WmfA0_0h9VPR70cQM3fBQkQm6g8RCnJ0VlZpxD_o7eAmSl9Qosg7mGQ8087HRt8GheE8aK3XbhMdU8G01PUk-tpwzuZ_VJrhHx0r-9tazBCbz0ELHs77UVs/s1600/DSC_0335.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrBcWmIKsNm70KhVpzhbI7WmfA0_0h9VPR70cQM3fBQkQm6g8RCnJ0VlZpxD_o7eAmSl9Qosg7mGQ8087HRt8GheE8aK3XbhMdU8G01PUk-tpwzuZ_VJrhHx0r-9tazBCbz0ELHs77UVs/s640/DSC_0335.JPG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Final assembly, left side</td></tr>
</tbody></table>
<br />
<br />
<br />
The resistors were soldered so they can easily be replaced. After taking some measurements I've decided to replace the resistors to a 10K fixed and a 15K trimmer instead of using 6.8K.<br />
<br />
I've also decided to use <b>two</b> 9V batteries in series as this improved both the response magnitude somewhat.<br />
<br />
I've simply poked a hole on the bottom of the tin can, and glued the PC board to it. I've soldered a 2.5cm / 1 inch stiff wire protruding from the board inside the chamber.<br />
<br />
Most tin cans are coated with an epoxy resin in the inside, so I needed to grind that away with a piece of sandpaper. Don't make the mistake I did and do this before assembly. ;)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbrFpZZScuqsc9phFpyEHXWC8f4qTCfqd1MVPmuyFSUfhMh5bv194GNy5sbH5ZJITLi0aBPT11f5jjMcT0ZpFIRnireYRtodxLlxGETUopIB0FqsQ3QKdmwAcOmRe8-e9MAO1T_OmpuP4/s1600/DSC_0336.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbrFpZZScuqsc9phFpyEHXWC8f4qTCfqd1MVPmuyFSUfhMh5bv194GNy5sbH5ZJITLi0aBPT11f5jjMcT0ZpFIRnireYRtodxLlxGETUopIB0FqsQ3QKdmwAcOmRe8-e9MAO1T_OmpuP4/s640/DSC_0336.JPG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The inside of the chamber with the sanded walls and elongated base lead</td></tr>
</tbody></table>
<br />
<br />
The chamber must be shielded and grounded to prevent static electricity to interfere with the readings. If you're lucky and can get a really thin aluminium foil, it <b>won't</b>
block even alpha radiation completely. According to my experiments the
foil I use blocks around 75% of alpha radiation, which fine I'd say,
because alpha is very energetic and it's very easy to detect with this
sensitive circuit.<br />
<br />
You can use hair-thin wires and solder it on the chamber forming a mesh. This will effectively shield static too, but won't prevent dust, humidity and UV light to get into it. All of these can generate false readings. <br />
<br />
<b>Turning it on for the first time</b><br />
<br />
Connect a power supply (or battery) and measure the current uptake with a DVM. It should be around 100uA. Anything between 20uA-200mA is possible and acceptable, and this depends on transistor types and also the specimen used.<br />
<br />
Double check that the chamber is shielded and is grounded. Let it sit for two minutes.<br />
<br />
Measure the voltage between the appropriate leads of the resistors and ground. You should get two nearly identical values, and preferably around half the voltage of your power supply. A voltage drop of a few tenths of volts will provide a good range and decent sensitivity.<br />
<br />
If the voltage is too high, use larger resistors, if too small, use smaller ones. If the voltage is too dissimilar (more than a few dozen millivolts) it either means your transistors are not properly paired up, or something conducting got onto one of the leads. This can be flux, fingerprint or glue. You might be able to remove such stains with isopropyl- or ethyl alcohol, acetone or petroleum ether.<br />
<br />
Measure the voltage between the appropriate leads <i>of the two resistors.</i> Now you're practically measuring the <i>difference</i> of the voltage on the resistors. This should be reasonably close to zero, and you should be able to set it to near zero with a trimmer.<br />
<br />
You'll never get a perfect zero, and the circuit will drift around a bit, but not nearly as much as a simple one would do. A few millivolts of drift is perfectly fine. I got about 3mV drift in the course of.... like... ever. :) I consider this circuit to be rock solid. :)<br />
<br />
It's also worth mentioning that although the circuit doesn't drift much, large jumps of a couple (dozen) millivolts can be observed a few times every minute. This can be from Radon decay that is present in the air naturally. This source of error can be cancelled if the chamber is sealed and left alone so most of the Radon decays away.<br />
<br />
Even after such sealing and "ageing" the chamber might be a bit "jumpy" and large spikes might be reported by the DVM. This can be because of gamma or beta rays are hitting the transistor junctions, or because of cosmic background radiation. Muons coming from above can easily hit the camber and cause a nice trail of ions causing these spikes.<br />
<br />
It'd be interesting to build several chambers plus a coincidence detector, and having a very own cosmic ray observatory. Truly fascinating!<br />
<br />
<b>Measuring samples </b><br />
<br />
This detector provides ample sensitivity. It can detect anything from my small collections of radioactive materials. Thoriated tungsten welding rods, thoriated lantern mantles give very nice readings even through aluminium foil.<br />
<br />
Alpha sources are more easily measured through a thin wire mesh with large holes. Alpha particles cannot penetrate more than a few centimetres / inches deep into air, and almost impossible for them to go through any solids or even liquid films. A thin aluminium foil is almost impenetrable for them.<br />
<br />
The ion chamber is the most sensitive to alpha and muons. Beta rays also leave thin traces of ions after them, so they can be detected too, but most of their energy will be absorbed outside of the camber, or in the chamber's walls, and not in the air. This energy is lost and invisible for our chamber. Gamma rays hurtle right through the chamber leaving almost no sign of them. They are hard to detect, and only large fluxes of gamma photons can be detected.<br />
<br />
My entire collections of materials packed closely together and inside a thick plastic box causes about 10mV of deflection. (Alpha is utterly blocked, a few beta electrons might escape, but most of the gamma radiation is glowing right through the walls of the plastic box.)<br />
<br />
Weak samples can be placed inside the chamber which is sealed afterwards. Readings can be monitored over a length of time and an average reading may be calculated.<br />
<br />
Be careful not to contaminate the chamber, and clean it thoroughly after placing anything inside.<br />
<br />
<b>Future improvements</b><br />
<br />
The chamber could be supplied with high voltage to gather ions more efficiently. Proper data logger circuit may be added. Several chambers could be paralleled and the outputs could be fed into a coincidence detector to build a muon observatory. :)<br />
<br />
We'll be back at these soon.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-58072261616194491222016-07-11T22:35:00.004-07:002016-07-11T22:35:46.283-07:00CUDA accelerated linear algebra with Python and TheanoTheano is a Python module that enables one to construct mathematical expressions with matrices and/or tensors (basically more than 2 dimensional "matrices").<br />
<br />
<br />
These expressions are than can be evaluated using Python, but Theano can translate the expression into a C program and compile it to binary. This way it can achieve respectable performance.<br />
<br />
<br />
But wait, there's more! Theano can build the program so certain - or all - parts of it run on a GPU. Yes, on your video card. Modern cards can do calculations in a way that makes them especially fit for doing linear algebra and similar operations. In "similar" I mean the execution of simple operation on lots of data in parallel. A GPU-s can be several (tens or hundreds of) times better, than your CPU.<br />
<br />
<br />
I'm going to show you how to exploit an NVIDIA GPU, using Python.<br />
<a name='more'></a><b>Dependencies</b><br />
<br />
On Linux - Ubuntu Xenial (16.04 LTS) - every good story starts with an "apt-get install". Our case is no exception.<br />
<br />
First, we'll need to add so:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1504/x86_64 /" > sudo tee /etc/apt/sources.list.d/cuda.list</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo apt-get update </span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo apt-get install python-numpy python-scipy python-dev python-pip python-nose g++ libopenblas-dev nvidia-cuda-toolkit gfortran</span><br />
<br />
It's a good idea to install the cuDNN libraries. These are routines for neural networks, but might be useful in other applications too. Theano can check cuDNN availability and make good use of it.<br />
<br />
To get cuDNN, you need to register at NVIDIA, and download the package manually. Both can be done here: <a href="https://developer.nvidia.com/cudnn">https://developer.nvidia.com/cudnn</a>.<br />
<br />
If all is well, you can download two .deb file for libcudnn5 and libcudnn5-dev (libcudnn5_5.0.5-1+cuda7.5_amd64.deb and libcudnn5-dev_5.0.5-1+cuda7.5_amd64.deb in my case, and at this time). Let's install them:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo dpkg -i libcudnn5_5.0.5-1+cuda7.5_amd64.deb</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo dpkg -i libcudnn5-dev_5.0.5-1+cuda7.5_amd64.deb</span><br />
<br />
You need to reboot your system to proceed. This is needed so the CUDA enabled drivers and libs will be loaded.<br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Let's install Theano itself:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">$ sudo pip -I install Theano</span><br />
<br />
<span style="font-family: inherit;">I've included the -I option so if you already have Theano installed for some reason, it will be re-installed. Internet wisdom says this might solve compilation problems.</span><br />
<br />
<span style="font-family: inherit;">Theano depends on numpy and scipy, these will be installed to. To not clutter your system packages all up, it's recommended to use a virtualenv. I might, or might not re-write this tutorial to do that.</span><br />
<br />
<span style="font-family: inherit;">On Ubuntu 15.04, this concludes the installation. However on 16.04, Theano must be patched.</span><br />
<br />
<span style="font-family: inherit;">Edit /usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/nvcc_compiler.py and add the following line between os.chdir(location) and p = subprocess.Popen(cmd, ...), near line 360:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">cmd.append('-D_FORCE_INLINES')</span></span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">This magic is needed for Theano to work with gcc 5.4.0, the default gcc on U<span style="font-family: inherit;">buntu 16.04. Without this, you'll get an error message:</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">...<br />/usr/include/string.h:652:42: error: ‘memcpy’ was not declared in this scope<br />...</span><br />
<br />
<span style="font-family: inherit;">shortly after that Theano will tell you that "CUDA is installed, but device gpu is not available". Bummer.</span><br />
<br />
<span style="font-family: inherit;">Well, this monkey-patch fixes this. Just make sure you re-patch Theano if you upgrade it and get the error message again.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Great, you can run the test program at <a href="http://deeplearning.net/software/theano/tutorial/using_gpu.html">http://deeplearning.net/software/theano/tutorial/using_gpu.html</a> to see if Theano can really use the gpu:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python theanotest.py</span><br />
<span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">Using gpu device 0: GeForce GTX 970 (CNMeM is disabled, cuDNN 5005)<br />[GpuElemwise{exp,no_inplace}(<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]<br />Looping 1000 times took 0.531866 seconds<br />Result is [ 1.23178029 1.61879349 1.52278066 ..., 2.20771813 2.29967761<br /> 1.62323296]<br />Used the gpu</span></span><br />
<br />
<span style="font-family: inherit;">Great!</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Happy computing!</span>Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-77089287108485125582016-07-11T17:16:00.000-07:002023-11-16T21:31:16.682-08:00Building a radiation detector from scratch 2 - the first detectorThe detector will consist of a tin can, two transistors and a DVM. I will also show how to improve and stabilize the basic circuit by adding an extra transistor and an identical "dummy" circuit to cancel leakage current and temperature dependence.<br />
<a name='more'></a>The circuit will be same as the one on this video: <a href="https://www.youtube.com/watch?v=96bybrgI6V0">https://www.youtube.com/watch?v=96bybrgI6V0</a><br />
<br />
An ion chamber is "just" a very sensitive resistance (or conductivity) measuring device. It measures the resistance of a gas, in our case, thin air.<br />
<br />
Air is not a terribly conductive medium. It's resistance is somewhere between 1.3*10<sup>16</sup> and 3.3*10<sup>16</sup> ohmmeter (<a href="http://chemistry.about.com/od/moleculescompounds/a/Table-Of-Electrical-Resistivity-And-Conductivity.htm">http://chemistry.about.com/od/moleculescompounds/a/Table-Of-Electrical-Resistivity-And-Conductivity.htm</a>). 10 volts only causes 8*10<sup>-14</sup> to 3*10<sup>-14</sup> Amperes, or 8-3 femtoamperes. We need a device capable of measuring such tiny currents. Actually since we'd like to detect radiation, we will be dealing with much higher currents, in the picoampere (10<sup>-12</sup>) range. With hot samples, currents can go up to tens of nanoamperes (n*10*10<sup>-9</sup>). (<a href="http://www.teralab.co.uk/Experiments/Ion_Chamber/Ion_Chamber_Page1.htm">http://www.teralab.co.uk/Experiments/Ion_Chamber/Ion_Chamber_Page1.htm</a>)<br />
<br />
A DVM, or digital voltmeter can measure milliamperes, or 10<sup>-3</sup> Amperes. We still need to get about nine orders of magnitude gain.<br />
<br />
We can quicly get an other 4 by cheating, and using the DVM's voltage measuring function to actually measure tiny currents. The DVM has an intrinsic <i>internal resistance</i>, often 10 megaohms, or 10<sup>7</sup> ohms. If the DVM can read <i>millivolts</i>, that means it will read 1mV if we send 10<sup>-10</sup> amperes through it. That's not terribly smaller than our target of 10<sup>-12</sup> - 10<sup>-15</sup> amperes!<br />
<br />
We will jump the gap with two appropriately connected transistors. Meet the <i>Darlington pair</i>.<br />
<br />
To build a Darlington pair, connect two transistors together by their collectors, and connect one emitter to the other's base. The circuit with the two connected collectors, the unconnected emitter and the unconnected base leads forms a device similar to a single transistor, only it's DC current amplification constant equals to the two transistors' same parameters <i>multiplied.</i> <br />
<br />
To build a good Darlington pair for our purposes, I recommend using cheap, generic bipolar transistors with <i>large</i> "hfe" or "beta" values. The BC548-C device for example is a good candidate, it has a hfe around 800. That is, a typical BC548-C transitor will allow a collector-emitter current 800 times larger to pass if you sent a small base-emitter current through it. Connect two of these puppies into a Darlington chain, and you get 160000, or one hundred and sixty <i>thousand</i> times the current out than the current in.<br />
<br />
Fix the darlington pair near the hole on the bottom of a tin can. Solder the collectors to the can. Pass the base lead through the hole. You probably want to solder a stiff wire a few cm long to the base lead. It should be at the center of the tin can NOT TOUCHING IT AT ANY TIME.<br />
<br />
Connect a 9V battery's positive lead to the can. Between the emitter lead and the negative battery lead should go your DVM in voltage measurement mode.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJlg7peiDnAE5Ocv4ngzHYPjgVps44NeP0GWTXV0frPAAPgCzpKFbig5vwvtEHtgafu34WQbebysf54wDhkSiZefbPThIrVf0jY8dzUTdo3jFomk-nR5OjJAb-12GMAZAdUWtJW81uupM/s1600/simple_ion_chamber.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJlg7peiDnAE5Ocv4ngzHYPjgVps44NeP0GWTXV0frPAAPgCzpKFbig5vwvtEHtgafu34WQbebysf54wDhkSiZefbPThIrVf0jY8dzUTdo3jFomk-nR5OjJAb-12GMAZAdUWtJW81uupM/s320/simple_ion_chamber.png" width="320" /></a></div>
<br />
<br />
<b>There you go, your first ion chamber!</b><br />
<br />
After hooking it up, wait a minute or so to let the DVM reading to stabilize. Carefully move a piece of radioactive material near the tin can. The radioactive radiation will generate ions that will be driven towards either the can or the naked Darlington base. A tiny-tiny current will flow. The transistors will amplify this current, which will pass through the DVM. The high internal resistance of the DVM will cause the still-small current to cause a significant voltage value to be shown. Congratulations, you've just measured some radiation. Cool.<br />
<br />
<b>Could it be even better?</b><br />
<br />
You could connect 2-3 9V batteries in series, to improve performance. You should get sharper, larger readings with more batteries. Don't add too much though, after maybe 5 batteries you risk shocking yourself!<br />
<br />
<b>Drift and zero reading</b><br />
<br />
Transistors are nasty devices. First of all, they are not linear, meaning they do <i>much more </i>than just multiply their base current with a constant (although that's a good approximation for a number of applications). Their parameters are also dependent on their temperature. This simple detector doubles as a temperature sensor. Which is not good at all. Readings will jump around wildly even if you touch the them for a few seconds. Background radiation levels are also hard to establish and/or compare, since the transistors have a significant <i>leakage current</i>, they allow a small current to flow across their C-E leads even if no current is flowing across their base.<br />
<br />
The leakage current AND the temperature dependence be greatly decreased by comparing the current in two identical circuits, one that measures radiation, the other measures nothing. When the temperature changes, the current also changes in both circuits in a very similar manner. The only difference is the radiation, and that can be measured by subtracting the leakage current.<br />
<br />
The next circuit will work like this:<br />
<br />
<ul>
<li>Build two <b>very similar</b> amplifiers, that are <b>thermally coupled</b>, meaning they are very close together, connected by a medium that can transfer heat well, so their temperature will hardly differ.</li>
<li>Make the current from each amplifier flow through a pair of identical resistors connected to ground.</li>
<li>Measure the voltage between the <i>hot</i> (non-grounded) lead of the resistors</li>
<li>The change in difference will be proportional to the change of conductivity in the ion chamber. </li>
</ul>
The resulting circuit will be surprisingly stable. With an extra part or two, it can also be zeroed, e.g. it can be adjusted to show (near) zero at normal background. It is sensitive enough to be used with fairly low activity samples such as thoriated tungsten welding rods, rainwater, and to absolutely go nuts around decent specimens of radioactive minerals.<br />
<br />
This device can be used to indicate and compare the radioactivity of samples, and even to determine the type of radiation emitted by the them. Specimen with very short half lives can be monitored to see the exponential drop in activity (like in the case of fresh rainwater).<br />
<br />
It is perfect for home and school experiments, or for hunting radioactive material.<br />
<br />
Unfortunately the device is probably pretty much non-linear, and most likely would take a complicated extra microcontroller circuit that would correct this error, and a lengthy process of calibration. However, once done, the controller could take the temperature, humidity and driving voltage into account to make a decent guess and bring this device close to a real radiation measurement equipment.<br />
<br />
There are other places of improvement too. The voltage applied across the ion chamber could be raised by more than an order of magnitude, to something like 400V. This change would make the device faster (respond with sharper pulses), and more sensitive, since ions and their electrons would be ripped apart by the large voltage potential, preventing recombination, and pulling them quickly to the electrode and the wall.<br />
<br />
<br />
I'm considering to do both changes to the current design.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-3535968492570544462016-07-11T16:51:00.002-07:002016-07-11T16:51:40.526-07:00Musings on the ADIF file formatThe ADIF file format is a simple way to store and organize Amateur Radio (or HAM Radio) log data.<br />
<br />
It is a way amateur radio operators most frequently store their records on the contacts they make (probably after paper).<br />
<br />
(For fellow HAMs, my callsign is HA5FTL, see my QRZ.com page for a description of my station and yours truly, and for contact information, if you wish to express your objections and/or approval of one or several of my points made in this article.)<br />
<br />
<br />
I have to start with a disclaimer.<br />
<br />
<a name='more'></a><br />
<b>The Disclaimer</b><br />
<br />
<br />
I have to premise that I respect the devotion and
countless hours of hard and precise work of my fellow Amateur
Radio Operators who proposed, devised, specified the ADIF ADI/ADX file formats, and kept it up-to date throughout all these years.<br />
<br />
I do not claim their work was in vain, or it's product is <i>really</i> unsuitable or too inferior to be the practical and enabling tool it really is.<br />
<br />
It's just that it could be much better.<br />
<b> </b><br />
So let me try to explain my expressed anger with this file format, and do not take it personally when I bash on it.<br />
<br />
Because I will do that a lot.<br />
<br />
<b>Why do that? - An introdction for non-HAM programmers</b><br />
<br />
Amateur radio operators are folks who are allowed to operate radio <i>transceiver</i> (a device that can receive AND transmit radio frequency energy) manufactured, modified (or hacked together) for being used by them for a great variety of purposes in greatly many ways. Not only that, they are allowed to operate transceivers that cover a HUGE portion of the whole radio frequency spectrum, down from a few kilohertz (well into audio frequencies), up to "daylight", very, very high frequencies.<br />
<br />
The exact frequency bands, usable power, and the way radio waves are transmitted all subject to strict regulations. (They cannot jam television or broadcast radio, military & defence frequencies, etc. basically common sense most of the time). An Amateur Radio License is special, because it grants freedom no commercial radio license can ever do.<br />
<br />
With great power comes big screw-up potential. HAMs have to pass not-so-easy <i>exams</i> to be given <i>licenses</i>, but after that - depending on the type of license - they can even <i>build</i>
their own radio receiver and/or transmitter (which can be surprisingly
simple, (or mindbogglingly difficult) so it's a great source of fun
& experience). The radio frequency spectrum is a very tight resource, a lot of player hope to grab a slice. Governments sell these slices at high prices, except for hams: they are given a lot of frequencies essentially free. All they have to do is to tightly follow the rules and cooperate with the authorities if requested.<br />
<br />
For governments to be able to monitor their HAMs (the operators) closely, all of them are REQUIRED to keep a record of all contacts (sometimes even attempts) made. The details vary across countries and circumstances these contacts are made or attempted.<br />
<br />
Operating an amateur radio station is a wonderfully diverse hobby, with wonderfully and surprisingly diverse bunch of people participating in it. From bright-eyed young girls to old, fat granddads, from musicians to royalties a lot of people can and do find pleasure in random or scheduled radio contacts, special events, local contests, or world championships.<br />
They buy, modify or build their equipment, and talk, use Morse code, or digital data transfer modes, operate a network parallel to the Internet, make contacts to and through the International Space Station, they use repeaters, satellites, stratospheric balloons, or using the ionosphere or even the Moon to bounce their signals toward lands beyond horizon.<br />
<br />
To engage in the hobby is truly an amazing experience. It is like having several interesting hobbies all at once. One can always find new, interesting things to achieve or discover. <br />
<br />
All these different people with different fields of interests are required to keep records of their contacts, and there would be several good reasons to do so even if we wouldn't care about the regulations. (We do, and very much so.)<br />
<br />
To make a lot of contacts, or interesting ones is a great achievement. Prestigious awards are given for those making contacts with the greatest number of operators in distant countries, on islands, or summits, or contacting a station operating only for a short period of time.<br />
<br />
Such awards can be applied to by handing in the records of the contacts: the logs, and most often by providing additional piece of evidence (such as logs, or written notes (QSL cards) from the contacted stations).<br />
<br />
<b>So what is this ADIF thing?</b><br />
<br />
ADIF stands for Amateur Interface Data Format, and it's the most popular way to send and often to store data related to HAM activities. In practice this is almost always means information about contacts made.<br />
<br />
<b>The Good...</b><br />
<br />
- ADIF is a standard.<br />
<br />
When it comes to exchanging data, any kind of standard is better than chaos.(NASA was taught a very expensive lesson on this).<br />
<br />
- ADIF is plain text<br />
<br />
You can open it in your favourite editor, and make changes. This IS a huge plus, as no program is perfect - you can resort to a generic editor should you ever need a transformation your logger application does not support. "Use text, because that is an universal interface" - stands in the UNIX philosophy, for very good reasons.<br />
<br />
ADIF is really just a collection of <i>records</i> that describe something, most often contacts between stations. A records has <i>fields</i>, that has a <i>name</i> and MIGHT contain data. A record is basically a set of key-value pairs. Pretty nice.<br />
<br />
For me the list pretty much ends here.<br />
<br />
<b>...the Bad and the Ugly</b><br />
<br />
I'm going to try to
explain this in a way that I hope brings my seemingly subjective spleen-spitting closer to be an objective and constructive criticism, <b>and hopefully point towards a good solution.</b> I'm going to be passionate, because this is what really drives me at this hour. <b> </b><br />
To be able to explain why am I so overpoweringly repelled by the standard, I'll try to assemble a short list of facts and explanations.<br />
<br />
- ADIF is a standard. And it is a BAD one.<br />
<br />
This sounds like circular reasoning: it's bad, because it's bad AND popular. I'll nevertheless keep this point, because popularity makes it even worse from a perspective I'm going to take later. For now, let's state that if something is bad, and it's the only path ahead, you have to stumble your way along it.<br />
<br />
<b>The solution for this is either to vastly improve the current one, or</b> to throw it out all together, and use a different format - probably an application of a general format - that is accepted widely. Yes, I'm thinking about <b>XML</b>. No, the ADX file format described in the ADIF specification is not XML. (More on that later.)<br />
<br />
- ADIF steals programmer's time <br />
<br />
That's that particular perspective I'll take now. From this it can be seen how an insufferable piece of software or standard can turn into something objectively ineffective.<br />
<br />
ADIF is BAD, because w<b>orking with the format is a huge cognitive load. It's hard.</b> It has quite a few catches, AND lacks widespread, well-tested programming libraries that work out-of-the-box. If you're writing a program that eats or spits ADIF, you're likely to be forced to roll your own parser or serializer.<br />
<br />
Yes, there are a few github projects, there are blogs on how to parse the darn thing in PHP and other languages, but it's not at all "import antigravity". <b>Dealing with ADIF takes time, and it takes a lot of time.</b><br />
<br />
If one uses their time to write ADIF juggling routines, they will have less time and energy for being creative, developing useful features or good-looking and convenient user interfaces.<br />
<br />
It also prevents small "fire-and-forget" projects to pop up, because no one can write an ADIF library in 30 minutes, and call it anything near complete or usable (trust me, you can't). There are no "do one thing, do it well" (another piece of UNIX philosophy) tools and shims with narrow, well-defined and tested functionality.<br />
<br />
The reason for it is the same as for there are no good libraries really. And this reason is the following:<br />
<br />
- ADIF is Hard and Rigid<br />
<br />
This is the reason why there's little to no available "canned" software for dealing with ADIF.<br />
<br />
The ADIF specification is a big document. Fully understanding and implementing it takes a lot of time. It defines a plethora of data types. A lot of these data types are <i>enumerations</i>, a list of possible values (words) that can stand at particular places in the file or data stream.<br />
<br />
One problem with this, is that it rigidly fixes a number of things that should be flexible. A program or library closely adhering to the ADIF specification SHOULD NOT accept a piece of data if it doesn't fit into the given data type. <br />
<br />
<br />
Let's take the MODE field for example.<br />
<br />
Of course any general program or library MUST follow the standard, or it's loses the meaning for it's existence. This means the proper validation of fields, and forbidding any new modes, contests, countries or subdivisions, bands, propagation modes, QSL mediums, etc. to be quickly adopted by the community.<br />
<br />
If you develop a new digital mode for example, there will be a significant resistance before it's acceptance, because any existing logger software will decline log entries with your new digital mode. <b>It is currently impossible to properly log contacts in the FSQ digital data transmission mode, because FSQ is not a member of the Mode Enumeration, and log entries containing "FSQ" in the MODE field are declined by logger programs and sites. </b>This clearly holds back FSQ and similarly new digital modes from being accepted. The less stations you can work in a mode, the less likely you'll use it. The less likely a mode is being chosen a station, it's even less likely to be chosen by others. The highly non-linear dynamics of the spread of information transfer modes greatly magnify any such resistance.<br />
<br />
One solution could be to modify the standard to be more easily extended <b>and state this possibility clearly in the specification<i>.</i></b> The effort required to write a log entry with a new digital mode is by no means prohibitive, <i>but others HAVE TO accept it</i>, for even such a small effort as writing a log with a text editor have any point at all. Unfortunately the standard does not make possible, or at least doesn't allow it explicitly.<br />
<br />
One could say that ADIF parser implementations are to blame, and we could try to convince the ARRL to make LoTW more flexible and please accept records about contacts in FSQ (or other new) modes. But sadly the ARRL and LoTW (for example) NEEDS the precise mode information to be able to give credits for contacts, because the credits are give <i>per mode</i>: you can get a credit for a phone (SSB, FM, AM), CW (Morse code) or <i>digital</i> contact.<br />
<br />
How unfortunate that ADIF lacks tags that describe general facts about contacts, such as whether the contact was made using analogue or digital mode, whether it used frequency shift keying/modulation, amplitude/on-off keying/modulation, phase shift tinkering, uses single or multiple tones, whether the carrier is partially or entirely suppressed, etc.<br />
<br />
I bet a sufficiently general and widely usable set of fields could be added so they are in practice (or even in theory) can completely describe a transmission mode, and the current mode enumeration would be a mere convenience, a collection of abbreviations for a set of mode-descriptions with a tendency to appear together in practice.<br />
<br />
In case of DXCC entries, countries, subdivisions, SOTA summits, IOTA islands and similar list of possible values <b>there are authoritative information sources</b>, such as the ARRL's <a href="http://www.arrl.org/country-lists-prefixes" target="_blank">DXCC entity list</a>, numeric-, alpha-2 and -3 ISO-3166 country codes, and the official sites of the SOTA and IOTA awards. <b>Such lists take up most space in the ADIF specification. They shouldn’t. There are places these pieces of data should and do live.</b> Mirroring these and similar list are causing inconsistency between the specification, the authoritative sources and/or reality, and keeps new members of these lists from finding their way into the logs of fellow HAMs.<br />
<br />
In my opinion, these lists should be replaced with references to the authoritative information sources and the possibility to add new members should be left wide open. Where necessary, a set of fields with describing general properties should be added, to allow passing <i>information</i> (not just data) on yet-unknown situations.<br />
<br />
To cut it short: <b>ADIF is hard, because including the enumerations are tiresome and error-prone, and it is rigid be cause these MUST be included by any serious implementation.</b><br />
<br />
The lack of general, widely used libraries seem to be a piece of empirical evidence for this.<br />
<br />
- ADIF doesn't natively support Unicode (or at least does it poorly)<br />
<br />
The ADIF format is really two file formats. An older "ADI" file format and a newer "ADX". The former was probably modelled after HTML, the latter after XML (<b>ADX IS NOT XML despite what the documentation says</b>, more on the actual formats later).<br />
<br />
<b>The spirit of ADI is to use only ASCII characters.</b> I don't know if there are any programs actually enforcing this constraint, but <i>there can be</i>, and if you want a truly compatible implementation, you better not include characters with codes above 127. You have to stuck with the English alphabet, numbers and punctuations/control characters.<br />
<br />
Of course the specification <i>can</i> be interpreted so that any <i>byte</i> is allowed as data. (Restricting tag names to be ASCII strings is actually a good idea given ADI is a textual format. It is the most compatible encoding, usable with text presentation/editor software using ASCII, Latin-X, UTF-8 and several other encoding methods / code tables.) If any byte-string of proper length would be EXPLICITLY allowed after tags in ADI files, this problem would quickly be resolved.<br />
<br />
<b>Adding an <ENCODING:5>UTF-8 or similar tag into the ADI header would make this misery disappear at once.</b> Encoding is only important, if data is displayed. For data transfer, it's enough to know how many bytes you have, and what are those bytes. An ADI tag contains every piece of information for ADIF to be a truly flexible, binary Amateur Data Interchange Format.<br />
<br />
Because of the the <b>*_INTL fields</b> for storing UTF-8 encoded versions of a few fields <b>are unnecessary</b> (and are strangely only part of the ADX file format, which really wants to be XML, and says it's UTF-8 encoded). <br />
<br />
Actually there is a recent data transfer and storage format that bears uncanny resemblance to ADI. It is BSON, the "Binary JSON", an efficient binary data format used by the popular document-orinted database server MongoDB. BSON's "records" are called "documents". Documents have fields, with name, data type, and actual data. The length of the data is also stored in fields (but since it is entirely binary, and has no delimiters at all, the length of the field name is also stored, and no extra data is allowed between fields or documents). So this name-length-type-data based principle <i>can</i> work well in practice, "only" the details have to be gotten right.<br />
<br />
Of course one can say that the standard is old, however UTF-8 is a great way to store Unicode text data backwardly compatible with ASCII. UTF-8 is ASCII, if you stick to code points below 128.<br />
<br />
- ADIF stores the type of the data explicitly<br />
<br />
Now it's time to dwell into the actual file format. A piece of data stored in ADIF looks like this:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><TAG:11:S>actual data blah blah lots of extra<br />characters that do not matter.</span><br />
<br />
There is some <i>metadata</i> (data about data) and the actual data. Metadata is most often stored in <i>tags</i>, similarly to HTML. Except there are no closing tags. This is probably to save space, as most data is represented as short strings comparable to the length of text describing the tag containing the metadata. Since the length of the data is known beforehand it need not to be interpreted in any way, and no escaping or similar trick is necessary. (As it was said above, ADIF can store opaque binary data.)<br />
<br />
Between < and > characters there MUST be a tag name, like CALL if the data is a <i>call sign</i>, or RST_SENT if the data is the signal report sent to the other station. If the tag does not store any actual information, than the name alone suffices. (Such tags are the <i>end of header</i> and <i>end of record</i> tags: <EOH> and <EOR> respectively).<br />
<br />
If the tag do contain data, the length of the data represented as an ASCII character string MUST be given after the tag name separated from it by a colon (:) character, and written in decimal ASCII digits, in base ten. After the length there MAY be a character - again, separated by a colon from the length - that denote they <i>type of the data</i>.<br />
<br />
This feature makes it simple to quickly write a parser that just "explodes" the ADIF string to (<TAG>, data) pairs. Just look for the first <, then read the tag name, length, and possibly the data type, than expect a >, and read the right number of bytes. Repeat until the end of the file.<br />
<br />
Unfortunately ADIF, as described here, cannot be cut up into tokens with the usual way (using regular expressions), as - strictly speaking - <b>the language is not regular</b>. This looks like a minor annoyance not really deserving it's own point. The proof would probably be possible by proving that recognizing this language would be "just as hard" as recognizing the language containing only words with the same number of a-s and b-s in it. The problem with this is that it is impossible to write a fast tokenizer with the popular tools (like lex of flex) that can provide high-level tokens and therefore require little logic above the token level.<br />
<br />
Low-level tokens like the <, :, and > characters, length, data type and tag names can be recognized, but additional logic is needed to read the number of bytes. This small design feature prevents the use of popular parser generators, and promote DIY implementations that can be buggy and hard to maintain.<br />
<br />
Even if you can write this software quickly, you still have to deal with data types.<br />
<br />
This data type may be string, if the character is S, a date if it's D, time if it's T, and so on. The standard describes quite a few data types and their format.<br />
<br />
This is problem, because for many tags, there is really only one sensible data type to use. Extra effort needed, to validate the type marker.<br />
<br />
This sounds like nothing serious, but if you don't work for Microsoft and aim for full interoperability, you HAVE TO do this. In practice, no one really uses the type marker for frequently used, standardized tags. <b>The data type should always be obvious from the field name, and this is how the type of the data is communicated and determined most of the time in practice.</b><br />
<br />
- Awkward data types<br />
<br />
There are data types that are non-scalar: lists, even lists of key-value pairs (such as CreditList). Handling these types requires manual labour. A file format describing structured data should provide a small set of simple solution covering every possibility. Tags, and members of an instance of the CreditList both essentially describe key-value pairs. We have two solutions for a single problem. This leads to an unnecessary large lines-of-code metric. Not good.<br />
<br />
<b>XML of course could solve this, as it provides the tools for describing nested data.</b><br />
<br />
<b>What can be done?</b><br />
<br />
Every problem I can think of can be efficiently solved by using a format that is a proper, well-designed XML application.<br />
<br />
There are good XML parser libraries for any programming language I can think of. These are ready to be used, and are essentially free of bugs (or their bugs and quirks are well-known).<br />
<br />
Upon these libraries general-purpose ADIF/XML parsers could be built with relatively little effort. These libraries than can be re-used by application developers, liberating them from the burden of writing their own ADIF parser.<br />
<br />
Such generic libraries would work correctly with application- and user-defined fields living in their own namespaces. File format validation could be programmed in a few lines of code given the XML schemas of the core document and the extensions.<br />
<br />
Since the effort to actually implement such libraries would be reduced, providing reference implementation and test suits wouldn't be such an outlandish expectation.<br />
<br />
<b>ADX is not entirely an XML application.</b><br />
Okay, but ADX <i>is</i> XML. Or is it? Well, not so much.<br />
<br />
For one, namespaces are ignored, killing extensibility. Namespaces <i>are</i> the perfect solution for incorporating application- or user-specific data. The whole <USERDEF ...> and <APP> thing is unnecessary.<br />
<br />
<br />
A true, extensible, simple XML application would solve all of these problems. ADX is close, let's hope we keep heading that direction.<br /><br />
<br />
Folks, let's use XML like it's meant to be. Please.<br />
<br />
Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-85142111799470777552014-07-29T02:57:00.002-07:002014-07-29T03:02:33.145-07:00Ubuntu: running services inside a chrootSo I got a task the other day: install php 5.5.9 onto a fairly old (10.04.4 LTS "lucid") Ubuntu machine, without breaking the already existing prehistoric php, installed from the default repository.<br />
<br />
Also, compiling from the sources was out of question, since my colleges wanted to keep the update procedure as simple as possible.<br />
<br />
So how do I install a 14.04.1 LTS "trusty" package onto lucid without wreaking havoc among the packages?<br />
<a name='more'></a><br />
I don't.<br />
<br />
First I thought I might install lxc or similar light-weight virtualization solution, but for a single service it seemed to be an overkill. When I saw that the node already using almos all phisical memory alredy, it was completely out of question.<br />
<br />
So I decided to build a minimal chroot environment with debootstrap. Unfortunately debootstrap on lucid won't build you a proper (or any) trusty chroot. I tried to install the trusty deboostrap .deb file manually with dpkg, but <i>of course</i> it was unsuccessful. Apparently even the .deb file format is somewhat different between the two versions of the distribution, the package manager never managed to even decompress the archive.<br />
<br />
So I just decompressed it on my laptop (running trusty), and compied debootstrap over the lucid machine.<br />
<br />
This was a complete waste of time, since the debootstrap script downloaded the .deb files OK, but - guess what - it could not decompress and install them properly. Surprise, surprise.<br />
<br />
My last chance was to build the chroot env on my desktop, and use that on the server (fortunately both are amd64).<br />
<br />
Here is the debootstrap command I used:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">debootstrap --variant=buildd trusty /jails/php559 http://archive.ubuntu.com/ubuntu/</span><br />
<br />
<span style="font-family: inherit;">This will download and install a very basic Ubuntu Trusty into /jails/php559.</span><br />
<br />
<span style="font-family: inherit;">One quirk is that you need to mount the /proc filesystem before doing any work in the chroot environment:</span><br />
<br />
<span style="font-family: inherit;"><span style="font-family: "Courier New", Courier, monospace;">chroot /jails/php559 mount -t proc proc /proc</span> </span><br />
<br />
<span style="font-family: inherit;">After this, you need to install packages like this:</span><br />
<br />
<span style="font-family: inherit;"><span style="font-family: "Courier New", Courier, monospace;">chroot /jails/php559 apt-get install <package name></span></span><br />
<br />
<span style="font-family: inherit;">As you've probably figured out, running anything inside the new environment can be achieved with</span><br />
<br />
<span style="font-family: inherit;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: "Courier New", Courier, monospace;">chroot /jails/php559 <command><span style="font-family: inherit;"> </span></span></span></span></span></span><br />
<br />
<span style="font-family: inherit;">Once you installed anything you needed, you can start / stop services inside the new environment. One thing to note is that those services and applications still use the kernel the node booted, so you can't run <i>too exotic</i> stuff inside a chroot, like a very ancient, or too new software, or software compiled for a different architecture. In those cases you need some kind of virtualization.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">One problem remains: how to start service inside a chroot?</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I wrote a very skinny init script that resides on the host node's /etc/init.d directory and blindly executes service <program name> <command> inside the chroot:</span><br />
<br />
<pre>#! /bin/sh
### BEGIN INIT INFO
# Provides: jail-php559-apache2
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: apache2 running in jail
# Description: Start / stop the apache2 instance with php 5.5.9
# in an appropriate Trusty chroot jail.
### END INIT INFO
# Author: Fabian Tamas Laszlo <tamas .fabian="" vengit.com="">
JAIL=/jails/php559
JAILRUN="chroot $JAIL"
SERVICE=apache2
case "$1" in
start)
# Ensure that /proc is mounted inside the chroot
$JAILRUN mount -t proc proc /proc
$JAILRUN service apache2 start
;;
*)
# Delegate any command to service command inside jail
$JAILRUN service $SERVICE $1
;;
esac
</tamas></pre>
<br />
Don't forget to run<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">update-rc.d jail-php559-apache2 defaults</span><br />
<br />
<span style="font-family: inherit;">after installing the init script.</span><br />
<br />Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-49033539450179269352014-06-28T08:20:00.002-07:002014-06-28T08:22:21.833-07:00Building a radiation detector from scratch 1 - the basics.<h2>
</h2>
<h2>
Before you begin</h2>
If you follow this series of articles and maybe do some research on your own, you'll be able to build a simple radiation detection device and perform some simple, interesting experiments.<br />
<br />
<span style="font-size: small;">I chose to build a simple, but surprisingly effective device called an <i>ionization chamber</i> or <i>ion chamber</i>, which is just a very sensitive current sensor, or resistance measuring device, the difference being only the point of view.</span> <br />
<br />
A great source of information on ion chambers is: <a href="http://www.techlib.com/science/ion.html">http://www.techlib.com/science/ion.html</a><br />
<br />
Ion chambers are <b>not</b> Geiger counters, however they work according to the same basic principles, as will later be explained. <br />
<br />
You will need to be able use soldering iron, and perform current and voltage measurement. If you don't know how to do these things, it's better to ask someone to show you. Please only perform the tasks written here if you know what you are doing. Soldering requires high temperature. We will work with high voltages at some point, and of course, there will be some radioactivity involved. Any of these things alone can be dangerous, so exercise caution and use common sense. Hurt yourself, blame yourself.<br />
<br />
To really build from scratch, one needs to understand the basic
principles on which the device operates. I'll describe these basics in
two articles, this one will be on nuclear radiation itself, the second
will be on basic electronics.<br />
<br />
This series of articles should be considered loose notes on these subjects, and will only contain material
vital for our purposes that is, building radiation detectors. If you don't understand something, do your research, or ask in the comments.<br />
<br />
<br />
<a name='more'></a><br />
<h2>
Bill of materials in advance </h2>
I tried to keep the circuits and the overall project as simple as possible, using the most common resources available. Below are an approximate bill of materials. This might change as we go along, but will certainly be enough for the first one or two experiments.<br />
<br />
<ul>
<li>Transistors. I used BC547C types, because of their high gain, and their price, which is next to nothing. You will need 3 in the basic circuit or 6 for the more advanced one, but I recommend you to buy at least 50, since we will need to match a certain parameter of them to be able to get any reasonable stability. You can probably use any NPN bipolar transistors with current gain around 500 in a TO-92 package. We will connect three of these to achieve something around <i>1 to 2 hundred million</i> current gain.</li>
<li>Resistors. The actual value you might want to use will be between 1 KOhm (kilo ohm) and 100 KOhm (or 1K and 100K). 10K is a safe bet. These are also very cheap, you might want to buy 10 each from the following values: 3.3K, 4.7K, 6.8K, 10K, 15K, 22K, 33K. Ask for through-the-hole, 1/4 or 1/2 watts, metal film types. These are probably the cheapest too. Don't buy SMD (surface mounted) types, unless you <i>really</i> know what are you're up to :)</li>
<li>If you want to build the high precision version, you might also want to buy a trimmer potentiometer, a 10K will probably be all right. </li>
<li>A digital multimeter - Make sure it can measure millivolts too. One with a short circuit indicator will help a lot. Probably this one will be the most expensive item in this list. If you want to build a really cool looking device, buy a 100uA (microAmper) analog current meter too, and use it with a 1K-10K resistor in series, as later will be demonstrated.</li>
<li>A 78L12 integrated circuit device. It should look exactly like the transistors above (it has a TO-92 package), and needed to keep the voltage fairly constant around 12 volts.</li>
<li>Two 1uF (microFarad) capacitors, aluminium electrolytic, 50V, needed for the 78L12voltage stabilizer. </li>
<li>I'm planning to build a high voltage power supply, but I'm not completely sure about the exact design. It probably will be a Dickson charge pump driven by an astable multivibrator.</li>
<ul>
<li>For that, you'll need two more from the transistors above, they don't even have to be matched.</li>
<li>You'll also need 20 diodes, 1n4148 will be OK. </li>
<li>20 100p (picoFarad) capacitors, at least 50 volts.</li>
<li>one 10nF capacitor, 400V</li>
<li>This version might need a "guard ring", a 1-2cm long coaxial cable will suffice. The exact type of cable doesn't matter much.</li>
</ul>
<li>A soldering iron. Use low power type suitable for the components above. A 30W iron is a good choice. Choose one with a small, pointy tip (need not to be needle sharp though).</li>
<li>Solder. Use thin, flux core solder.</li>
<li>Empty tins cans, two of the same shape, circular, flat. Try to get ones with 8-12cm diameter. The ones I used contained tuna. When buying the cans, use magnets to check their material. You need cans that stick to the magnet (steel cans). Aluminium cans cannot be soldered (easily), and must be drilled and equipped with screws, washers & pins to create reliable electrical contacts. As long as you can get steel cans, don't bother with aluminium ones.</li>
<li>Some fine sandpaper for sanding the tin cans, since they are likely to be coated with a thin layer of plastic foil or paint. This need to be removed.</li>
<li>A sheet of thin aluminium foil (the thinner the better)</li>
<li>Some electrical tape / duct tape</li>
<li>Some wires</li>
<li>2 9V Batteries</li>
<li>2 Battery clips</li>
<li>Some source of radiation for testing - you might already have some ;) </li>
</ul>
<br /><h2>
What is <i>radiation</i>?</h2>
<br />
The word "radiation" can mean a lot of things depending on the context. It is used in many scientific fields and pseudo-scientific settings. For our purposes the word <i>radiation</i> will mean <i>nuclear radiation</i>, the kind of energy emanation that happens as a consequence of <i>nuclear processes</i>, or those coming from the nuclei of atoms.<br />
<br />
We will take a close look the three most common types of nuclear radiation. These three share an important propery: they are all a kind of <i>ionizing radiation.</i><br />
<br />
<h2>
The makeup of matter</h2>
To make any sense of the stuff above, you have to know a thing or two about the structure of matter, about how the everyday stuff you see is built from basic building blocks.<br />
<br />
If you've been through elementary school, you probably know that stuff around you and even inside you, is made out of tiny particles, called <i>atoms</i>. Though once thought to be the final, indivisible constitutents matter, atoms too are built from even smaller particles.<br />
First of all, every atom has an outer <i>cloud of electrons</i> loosely jiggling around the small, and very dense <i>nucleus</i>. The nucleus itself can be divided into particles called <i>nucleons</i> of two types: <i>protons</i> and <i>neutrons</i>.<br />
<br />
All the chemistry and electronics happens in the outer <i>electron cloud</i> of atoms. Nuclear decay and other nuclear processes such as fusion and fission happens inside the nucleus or between atomic nuclei.<br />
<br />
<br />
<h2>
Alchemy FAIL</h2>
<br />
One very important difference between the chemical and nuclear processes is the amount of energy involved. The electrons are really just very loosely around the nucleus relatively to how densely the latter is packed. This means that the amount of <i>energy</i> in the nucleus is much, much higher than in the electron cloud.<br />
<br />
This means that no chemical process can ever influence nuclear processes. This is why alchemists have never succeeded in creating gold by mixing any number of materials together. The energy levels in a beaker are simply much, much too low to possibly be able to influence the nucleus. Given that gold is an <i>element</i>, and has a unique nucleus, it would require changing one kind of nucleus into an other one. Neither ancient alchemy nor modern chemistry had or has any chance doing so.<br />
<br />
So it is utterly impossible to influence nuclear events with chemical
ones, it is quite possible and natural the other way around. Nuclear
processes do effect chemistry and electronics. This is what we will
exploit in our nuclear radiation detector.<br />
<br />
<h2>
Electronvolts</h2>
OK, we've talked much about energy levels, but never mentioned numbers, just "large" and "small". So, what counts a "large" and "small" energy at the atomic and subatomic level?<br />
<br />
First, we need a good unit for measuring energy. The SI unit is joule, which is useful for measuring energies in everyday life. But this unit is too big. A joule is the amount of energy that needed to move something with 1 newton force through a 1 meter path, or the energy you get when a coulomb charge passes through a resistor with 1 volt through it's pins. One would need to accelerate particles to relativistic speeds to achieve such energy levels.<br />
<br />
There is a nice, small, non-SI unit of energy that can be used for our purpose. This is the <i>electronvolt</i>, or eV. One eV is very small compared to joule.<span style="font-family: inherit;"> 1 eV = 1.602177*10<sup>-19</sup> J</span><br />
<br />
One electronvolt is the amount of <i>inertial energy</i> an electron has once it has been accelerated with one volt. If someone would build an electron tube and connect a single volt to it's accelerator plate, the electrons arriving at the plate would have one electronvolt inertial energy.<br />
<br />
Visible light comes in packets of energy called <i>photons</i>, with energies between 1.6 eV and 3.4 eV. Red photons have lower, blue photons have higher energy.<br />
<br />
When burning hydrogen, the amount energy released is 286 KJ/mol. For one molecule that is 286 / (6*<span style="font-family: inherit;">10<sup>-23</sup></span>) = 4.77*10<span style="font-family: inherit;"><sup>-22</sup></span> KJ = 4.77*10<span style="font-family: inherit;"><sup>-19</sup></span> J = 4.77 eV. Such energies are typical in chemistry.<br />
<br />
The energy of radiation particles are greater by a factor of <i>hundreds of thousands to millions.</i><br />
<br />
<h2>
So what nuclear radiation really is?</h2>
<br />
<br />
When an atomic nucleus decay, it turns into an other kind of nucleus. Yes, what alchemy failed to do, happens in nature spontaneously all the time. Unfortunately gold won't form just like that from any cheap stuff, so this fact won't make us magically super rich overnight (or at all). However, the <i>byproduct</i> of nuclear decay is a big burst of energy - nuclear radiation. There are three types of nuclear radiation we will consider here.<br />
<br />
<h3>
Alpha radiation</h3>
<br />
is a stream of very high speed helium nuclei, consisting of two protons and two neutrons. The inertial energy of these nuclei typically in the ballpark with that of an electron accelerated with a voltage of five million volts, or to put it differently, is around five mega electronvolts or <b>5 MeVs</b>. This is about two <i>million</i> times higher than the energy of a single photon of visible light (<span class="nowrap">1.6 eV</span> to <span class="nowrap">3.4 eV).</span><br />
<br />
Alpha radiation can be very harmful if the source gets into the body, but quite harmless otherwise, since alpha particles are stopped by a single sheet of paper, the dead skin cells on your hand, or a few centimetres of air. Alpha particles can only penetrate very thin materials, such as the gold lead in the famous <a href="http://en.wikipedia.org/wiki/Geiger%E2%80%93Marsden_experiment" target="_blank">Geiger-Marsden experiment (also called the Rutherford gold foil experiment)</a>.<br />
<br />
These effects are all because the alpha particles are heavy, charged, and interact quite readily with any kind of ordinary matter, so they deposit their energy into pretty much the first thing they bump into. They can easily rip the electrons from the electron clouds of atoms, and can even fuse with light nuclei (such as aluminium). The products of such fusion are often unstable, and themselves undergo radioactive decay, they can emit neutrons, protons, electrons or gamma rays (see below).<br />
<br />
<h3>
Beta radiation</h3>
Beta radiation is a stream of high speed electrons. They are produced together with a so-called anti electron-neutrino (wich is extremely hard to detect), and a proton when a neutron decays.<br />
<br />
The typical energy of a gamma particle is around <b>1MeV</b>, but it can be as low as a few KeVs, or as high as a few tens of MeVs.<br />
<br />
Beta rays tend to penetrate deeper into matter than alpha rays, so they are harder to stop. A few millimeters of aluminum, or a centimetre or two of clear plexiglas or polystyrene can stop most of the gamma particles.<br />
<br />
One dangerous property of beta radiation is that it can produce secondary <i>brake radiation</i>, or Bremsstrahlung. When a fast electron hits something, it decelerates and some of it's energy is emitted as a high energy photon. This is how X-Ray machines work: they shoot electrons at a metal plate, and the collision produces X-Rays.<br />
<br />
The heavier atoms the obstacle the electron hits has the more Bremsstrahlung is produced, so beta radiation is better stopped with materials containing light atoms: aluminium and plastics (carbon, oxygen, nitrogen, hydrogen) are good choices.<br />
<br />
As we already mentioned, beta radiation also a kind of ionizing radiation. These high-speed electrons can knock off other electrons from around atomic nuclei, much like alpha radiation, but with much less vigour.<br />
<br />
<h3>
Gamma radiation</h3>
Gamma radiation is electromagnetic radiation. It is the same kind of energy than light, or radio waves, only quite a bit more powerful. The typical gamma ray has a <b>few hundred KeVs</b> of energy, but can be much lower or higher than that. This is no way a definition, only a rule of thumb.<br />
<br />
Gamma rays interact more reluctantly with matter than alpha or beta rays, but (or because of that) they penetrate deeper. One needs several tens of centimetres of lead to "stop" gamma rays. Even with such shielding, gamma rays cannot be stopped 100%. You can get arbitrarily close to that, but never truly reach it, there’s always a tiny chance of one or two of them slipping through. In practice, a few large lead bricks can protect even highly sensitive equipment from being affected too much by natural background radiation.<br />
<br />
Gamma rays can interact with the electrons in atoms by giving them some of their energy probably knocking them off from their orbits. This phenomenon is called <i>Compton scattering</i> or <i>the photoelectric effect</i>.<br />
<br />
There are materials that can change their electronic properties when hit by ordinary, visible light because of similar effects. Such materials are used in digital cameras for example.<br />
<br />
<h3>
Other kinds of radiations</h3>
There are as many type of radiation as particles - one could even accelerate mercury nuclei and call it "mercury radiation". However, in nature, and for our purposes, these three kind of radiation is all we care about, but there are some other cases that I find worth mentioning.<br />
<br />
I already wrote about neutron and proton radiation, but these are relatively rare. Also beta radiation can consist of positrons (anti-electrons), but they will quickly annihilate with a regular electron producing (in the most common case) two photons with an energy of about 511 KeVs.<br />
<br />
There are some other exotic particles that do happen in nature, mostly because of cosmic radiation. The most common particle radiation at and around sea levels is the muon. This particle can put out quite a show in <i>spark chamber</i> (look one up at YouTube).<br />
<br />
<br />
<h2>
Detecting radiation</h2>
All detectors rely on the ionizing nature of nuclear radiation. Our solution will measure the conductivity of air. When air subjected to radiation, some of the air molecules will be stripped of an electron or two. The electrons and the resulting <i>ions</i> will drift away from each other if there's an electric field around.<br />
<br />
The conductivity of air is very, very low (it doesn't conduct electricity very well). Even high voltage wires hanging on poles won't cause much current to flow in the air.<br />
<br />
However, air can become quite conductive when ionized. Very high voltages can rip electrons off from the air molecules. Such high voltages than accelerate these electrons so quickly than when they encounter an other air molecule, they themselves can knock off more electrons from it. An <i>avalahce effect</i> takes place. This happens during thunderstorms, or most of the time when an arc forms.<br />
<br />
Ions can form in the air if there are energetic particles passing through it. If we measure the conductivity of air, we can guess the amount of radiation passing through it.<br />
<br />
<br />
<h2>
Ion chamber </h2>
<br />
We will build a metal chamber, and stick a wire inside it, than apply some voltage between the chamber and the wire probe. If we can measure the tiny current flowing between the probe and the chamber walls, we can see how much ions are in the chambers. Since the currents are in the femto- to picoampere range, we need to amplify the current a few million times.<br />
<br />
How can we do that? See the next article.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-22172138923062788712014-06-10T05:21:00.000-07:002020-06-18T03:50:56.303-07:00How does iptables hashlimit module work?<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.imgur.com/69xwOj0.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://i.imgur.com/69xwOj0.jpg" height="192" width="320"></a></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">*UPDATED*</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Hashlimit is an iptables module that allows one to define rules that <i>in effect</i> will limit traffic speed (bytes / time unit) or frequency (connections / time unit) per target or origin ports / IPs. The inner workings of this module and / or how to make it work correctly remains a mystery for many.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Hashlimit is also close friends with the <i>limit</i> module, only much more powerful, capable of expressing rate limiting per source IP (for example) in a single rule. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<a name='more'></a><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">A hashlimit rule doesn't work like the most people I met imagine. The common false picture of iptables hashlimit is that it throttles connections by "slowing down" things somehow. Why </span><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">would </span>anyone </span>think that? It's just an iptables rule after all: match, or no match.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">But hashlimit doesn't work like that at all. Instead of applying friction to the wheels slowing the wehicle down, it just blows the whole wheel up. Instead of "slowing things down", hashlimit drops packets (in the most common use cases).</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Well... a haslimit rule actually either <i>mathches</i> a packet or doesn't, just like any other iptables rule. <b>If a hashlimit rule matches a packet, it means that the packet is below (--hashlimit-upto) or above (--hashlimit-above) a certain rate (bytes / timeframe or frequency / timeframe).</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">You can, of course, create a rule that <span style="font-family: "courier new" , "courier" , monospace;">-j DROP<span style="font-family: "arial" , "helvetica" , sans-serif;"> packets that are <span style="font-family: "courier new" , "courier" , monospace;">--hashlimit-above 10/sec<span style="font-family: "arial" , "helvetica" , sans-serif;"> effectively prohibiting traffic faster than 10 packets per second.</span></span></span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This means it isn't inherently connection-oriented either. Yes, it <i>can</i> be used together with connection tracking, but it's not mandatory. TCP connection rate can be controlled with hashlimit by simply matching on the SYN flag.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Ok, so hashlimit can only DROP, or ACCEPT a packet, or do the usual iptably things like directing it to a separate chain where it gets logged and then dropped etc. But... </span><br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">How can this even work?</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I mean how can this smoothly limit traffic, not causing connection resets and other weird end-user-deterrent errors? </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">There seem to be a contradiction between the theory and practice. If hashlimit rules simply just drop packets then why the end user only feel <i>maybe some</i> slowdown? Why there are no error messages saying that the connection was terminated? (Or at least why such thing never happens if the connection limit is higher than a few / min?)<br /><br />The reason for this is because almost everybody uses TCP. On the web, folks use TCP exclusively. And those who don't, probably do the same thing as TCP:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div style="text-align: center;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.imgur.com/6nWE2Pw.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://i.imgur.com/6nWE2Pw.jpg" height="279" width="320"></a></div>
<br /></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The TCP protocol sends acknowledgements, so the sending party can keep track of what have arrived and what was probably-lost on the network, and can re-send data if needed.<br /><br />The typical use-case for hashlimit is to drop the first TCP segment (the one with the SYN flag set) if there were too many in the near past, effectively limiting connection rate. The sending party keeps retransmitting connection requests for a while. The exact timeout and the number of retries are operating system (and configuration) dependent.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">An unfortunate consequence of this is if you need to tune your hashlimit <i>tight</i>, you will get dropped packets all the time.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.imgur.com/OO1CPfY.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://i.imgur.com/OO1CPfY.jpg" height="215" width="320"></a></div>
<div style="text-align: center;">
<br /></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">I saw system administrators carefully analysing the syslog, and trying to tune hashlimit rules so no packet drop log entries appear when traffic is "normal", but still trying to hold off simple DOS attacks (like a stuck F5 key).</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHT4aoxtZGl2p2PZfhwip9y-QUmjxiV6UzA0a2Kg1z8oNU8j9l1EYcCR-_1TSSVfrHxGkzq6AB0MLKECwgctNHOcpEOjSJ2AOg66Bu0RvKaaGzCGtf2NQhXKrt5xcjK4RmcVlx8KNAuNg/s1600/truestory.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="273" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHT4aoxtZGl2p2PZfhwip9y-QUmjxiV6UzA0a2Kg1z8oNU8j9l1EYcCR-_1TSSVfrHxGkzq6AB0MLKECwgctNHOcpEOjSJ2AOg66Bu0RvKaaGzCGtf2NQhXKrt5xcjK4RmcVlx8KNAuNg/s1600/truestory.jpg" width="320" /></a></div>
<div style="text-align: center;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Needless to say, such efforts are a waste of time. No matter how hard you try, there will be packet drops with hashlimit, and it's OK. TCP can handle that.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Of course if you have to tune limits <i>that tight</i>, you also gonna have a bad time with NAT users too. If your application can't handle at least a few hundred req/s per thread, then in general you gonna have a bad time, and hashlimit won't do much for you. I had an</span><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> old client once, whose application started coughing at <i>five</i> requests per second. Hashlimit wasn't quite the solution they needed. </span>Such an application will crash an burn as soon as it is introduced to the general public.</span></div>
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Experimenting with hashlimit</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>*UPDATE* I received a great comment from Tomasz P. Szynalski's. Apparently, I had a mistake in the examples. Quoting what I think is the most important:</b></span><br />
<span style="font-family: arial, helvetica, sans-serif;"><br /></span>
<span style="font-family: arial, helvetica, sans-serif;"><i>The rule "iptables -I INPUT -m hashlimit -m tcp -p tcp --dport 80 --hashlimit-above 20/sec --hashlimit-mode srcip --hashlimit-name http -m state --state NEW -j DROP" is not doing what you think it's doing.</i></span><br />
<span style="font-family: arial, helvetica, sans-serif;"><i><br /></i></span>
<span style="font-family: arial, helvetica, sans-serif;"><i>Iptables modules are executed in the order they are given in the rule. Because in the above rule, "hashlimit" comes first, it will process EVERY packet, not just new TCP packets, and every packet will count towards the 20/sec limit.</i></span><br />
<span style="font-family: arial, helvetica, sans-serif;"><i><br /></i></span>
<span style="font-family: arial, helvetica, sans-serif;"><i>At the beginning, there is no difference, because all packets are for NEW connections, but later on, if you have an open connection that keeps sending packets, those packets will keep depleting the hashlimit counters. So you may not be able to open a new TCP connection, even though the hashlimit rate says you should be able to.</i></span><br />
<span style="font-family: arial, helvetica, sans-serif;"><i><br /></i></span>
<span style="font-family: arial, helvetica, sans-serif;"><i>Instead, you want to FIRST check if a packet is TCP, then check if it's for a NEW connection, and only then send it to hashlimit to be counted.</i></span><br />
<span style="font-family: arial, helvetica, sans-serif;"><br /></span>
<span style="font-family: arial, helvetica, sans-serif;">Thank you sir, I stand corrected. On with the examples:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let's take a look at the documentation: <a href="http://ipset.netfilter.org/iptables-extensions.man.html#lbAW">http://ipset.netfilter.org/iptables-extensions.man.html#lbAW</a> and set up a hashlimit rule.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">For example, we can create a rule that will DROP packets that are NEW, coming to the http server (at port 80), and are coming too fast. In this case <i>too fast</i> means 20 in every second. We will enforce this limit by source IP address (hence the --hashlimit-mode srcip). The hash table name will be "http", so we can use this very hash table in other rules too if we need.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<span style="font-family: "courier new" , "courier" , monospace;">iptables -I INPUT -m tcp -p tcp --dport 80 </span><span style="font-family: "courier new", courier, monospace;">-m state --state NEW</span><span style="font-family: "courier new", courier, monospace;"> </span><span style="font-family: "courier new", courier, monospace;">-m hashlimit</span><span style="font-family: "courier new", courier, monospace;"> </span><span style="font-family: "courier new", courier, monospace;">--hashlimit-above 20/sec --hashlimit-mode srcip --hashlimit-name http -j DROP </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Verify</span> that the rule is active by issuing <span style="font-family: "courier new" , "courier" , monospace;">iptables -L<span style="font-family: "arial" , "helvetica" , sans-serif;">.</span></span><br />
<br />
<pre>Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP tcp -- anywhere anywhere tcp dpt:http state NEW limit: above 20/sec burst 5 mode srcip
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
</pre>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(Yes</span>, it is. Also, please note that this will limit <i>outbound</i> traffic too, since SYN,ACK response packets are considered NEW.)<br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Burst?</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As you can see, there is a "burst 5" note in the output above. This is the default "burst" value for every hashlimit rule. This has things to do with how the whole limiting stuff is implemented.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">When a packet comes in and hits the hashlimit rule, and it also matches all the other criteria for that rule, the hashlimit engine kicks in, and tries to find an entry in a hash table for that packet. In our case, it will hash the source IP address, and store <i>a counter</i> at that hash value, so our rule will maintain a counter for every source IP address.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">That counter <i>decreased</i> upon each hit. If the arrival of a packet tries to push the counter below zero, it means that <i>we have just hit the limit</i>. In our case it means that the packet must be dropped. If there isn't any counter stored for the current hash, one will be created and initialized to 1, so the fresh packet will be granted access, and the counter will be set to zero immediately.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The counter is also <i>incremented</i> in certain intervals, and the interval depends on the limit. In our case, the limit is 20/sec, so the counters are incremented by one in every 1/20th of a second, <i>until it reaches the burst value</i>. A</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> counter can never be greater than the burst value.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Actually this description simplifies things a bit, but not too much. The truth is somewhat uglier than that. See the Linux kernel itself for the gory details: <a href="http://lxr.free-electrons.com/source/net/netfilter/xt_hashlimit.c#L382">http://lxr.free-electrons.com/source/net/netfilter/xt_hashlimit.c#L382</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Testing with ab</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let's run ab (Apache Benchmark) to see what we've achieved. You need to have an apache server running locally. The default installation and the default "It works!" page will do just fine.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace;">ab -c 1 -n 10 http://localhost/</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<pre>This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient).....done
Server Software: Apache/2.4.6
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 11 bytes
Concurrency Level: 1
Time taken for tests: 9.001 seconds
Complete requests: 10
Failed requests: 0
Write errors: 0
Total transferred: 2550 bytes
HTML transferred: 110 bytes
<b>Requests per second: 1.11 [#/sec] (mean)</b>
Time per request: 900.064 [ms] (mean)
Time per request: 900.064 [ms] (mean, across all concurrent requests)
Transfer rate: 0.28 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 899 315.9 999 999
Processing: 1 1 0.2 1 1
Waiting: 0 1 0.2 0 1
Total: 1 900 315.8 1000 1000
ERROR: The median and mean for the waiting time are more than twice the standard
deviation apart. These results are NOT reliable.
Percentage of the requests served within a certain time (ms)
50% 1000
66% 1000
75% 1000
80% 1000
90% 1000
95% 1000
98% 1000
99% 1000
100% 1000 (longest request)
</pre>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">But WHY??? 1.11 requests / second is totally not 20 requests / second. What isn't working properly?</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">TCP. And it's working just fine.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">One can try and attack the local web server on multiple threads, hoping that the effect of the TCP retransmission timeout will disappear once there are many parallel connections. Let's try this one:</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace;">ab -c 10 -n 10 http://localhost/</span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">So all ten requests are made at once. Still, we see only <b>5 requests per second</b>. How can that be?</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This is because ab will try to make requests on all threads at once, and there's no chance for the counter to be incremented by even one. As soon as you issue the command, almost all SYN packet will be dropped, and the threads will wait for their chance to retransmit.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> When they get their chance, the counter will be at the the burst value, and when the threads send their retransmissions, <burst> number of threads will be successful.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">One way to circumvent this is to either reduce the retransmission timeout, or wait a little between each requests. Unfortunately, retransmission behaviour currently can't be set per-socket under Linux. There is a patch for that however, so this feature might make it's way into the mainline code later.</span><br />
<br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Also please note that if we turn "keepalive" on, then nothing will hold ab back:</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace;">ab -c 1 -n 10 -k http://localhost/</span></span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">I
got a nice 4500 reuqests / second with this. Fortunately DOSers rarely
use keepalive, and it can also be turned off at the server side. Of
course one should carefully consider (and measure) the effect of
keepalive settings, it can have a quite heavy impact.</span></span></span></span><br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Testing with curl </span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The following piece of bash script can be used to experiment with hashlimit rules, and their effect on the traffic:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">while true; do curl http://localhost/ &> /dev/null; echo -n "#"; sleep 0.05; done</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif;">The 0.05 seconds sleep will correspond to 20 reqs/s (or a little less, because curl needs some time to run too). If you try to decrease this, eventually you will see that the flow of # characters is being interrupted for a short period of time every now and then.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">If you sleep to little, the second SYN packet will be dropped, and curl will have to wait to retransmit, limiting the connection rate down to 1/<retransmission timeout> reqs / sec in case of a single thread. This causes the above script to print one # character in every second.</span><br />
<br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Conclusion </span></h2>
<h2>
</h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">TCP is a very sturdy protocol and can be configured to fit many environment. You can use it between two processes on the same core, or you can put a device to an orbit around planet Mars and expect a connection to stay alive despite the 6 - 42 minutes "ping" and probably heavy packet loss.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Even the desktop-default TCP stack can deal with the packet loss caused by hashlimit rules, and this is the way hashlimit rules supposed to do their work, so tuning for no packet drops is an unnecessary and futile effort.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Despite the fact both TCP and hashlimit rules operate along quite simple principles, their interaction can produce fairly complex behaviour. The behaviour of a system with very simple rules can be arbitrarily complex. Just think about how simple rules define the Mandelbrot set.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Because of this, you should always be aware of how the tools you use actually work. No metaphors or graphic images-in-mind will be able to help you pinpoint the source of a complex (mis)behaviour like an ab test producing 1-5 reqs/s under a hashlimit 20/sec rule. You need to know the system works <i>exactly</i>, nothing else will work.</span><br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
</span></h2>
<h2>
</h2>
Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com4tag:blogger.com,1999:blog-4443103563245822857.post-58843791734539547182014-04-20T00:04:00.003-07:002014-04-20T00:04:49.062-07:00Firefox and testing concurrent web application<b>TL;DR:</b><br />
<br />
<b>firefox won't start a request for an URL if a request is already being executed for that same URL.</b><br />
<br />
<b>If you open http://example.com/foo, and it takes several seconds to complete, when you simultaneously open a new tab and enter the very same URL, the latter request WILL NOT BE SENT until the former has completed.</b><br />
<br />
<b>If you ever need to make parallel requests for the same URL make sure you put some random query string in them, so the browser won't serialize requests for them.</b><br />
<br />
I was testing a concurrent web application written in Haskell, Yesod. I love Haskell and the Yesod framework, this combination gave me the best web development experience so far. I especially loved how easy it was to develop a simple web-based chat application by using TVars and TChans - variables and "channels" - basically message queues - to store state in a Yesod web application and pass messages between threads.<br />
<br />
I stumbled into a strange bug, and it took a good two hours of my life.<br />
<br />
<a name='more'></a><br />
<br />
For those who aren't familiar with Yesod, just imageine a usual MVC-based web application. My problem was to block the execution of a thread executing a certain controller, and continue it when an other thread executing an other controller sends a message down a pipe (a TChan to be more precise). This makes long polling possible.<br />
<br />
This is easy to achieve in Yesod, since it is multithreaded by default, and it isn't particularily hard to set up the TChan and use it from the Handlers (controllers).<br />
<br />
But when I tried to test the application, the strangest thing would happen.<br />
<br />
I opened up a browser (Firefox) and entered the URL that should been blocked, and it did: the loading indicator spun indefinitely until the fairly long timeout has been expired.<br />
<br />
When I simultaneously opened the other URL for the message sending controller, the receiver URL loading completed immediately, and the message appeared on the screen.<br />
<br />
So far, so good.<br />
<br />
So let's open two receivers, and thest the broadcast capabilities of TChans!<br />
<br />
I opened up two receiver tabs, both started to load. Then I opened up a sender, and ONLY ONE RECEIVER got the message.<br />
<br />
I spent about two hours debugging, including digging into the Yesod codebase trying to figure out what was going on.<br />
<br />
Finally I realized that the two receiver executions are completely serialized, and this could not happen inside Yesod: testing with curl and ab showed that calls to the same Handler are correctly being served in parallel.<br />
<br />
Then it hit me. What if Firefox blocks the execution of requests for the SAME URL for - I don't know - preventing simple (intentional or unintentional) DOS scripts from running.<br />
<br />
Aaaand I was right. Adding a "?asdasd=asdasd" after one of the receiver URLs fixed the problem, the requests was made parallel and the broadcast feature worked like a charm.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-14584710898475511152014-04-08T14:52:00.004-07:002014-04-09T11:42:14.146-07:00How to setup a raspberry PI as an IPv6 router with a SIXXS tunnel<div class="separator" style="clear: both; text-align: center;">
</div>
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
IPv6?</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">Internet Protocol version 6 is a <i>network protocol</i> that will soon replace the current Internet Protocol version 4 - the protocol that runs the Internet, and has those familiar four-number addresses like</span><br />
<div style="text-align: center;">
<span style="font-family: Arial,Helvetica,sans-serif;">183.43.221.13</span></div>
<div style="text-align: center;">
</div>
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">With IPv6 are coming a plethora of changes, probably the most end-user-alarming one will be the change in the address format demonstrated above.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The four-times-one-byte address will be replaced by a eight-times-two-bytes address that looks like some fancy password from a bad computer movie.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The new addresses will take the form of</span><br />
<br />
<div style="text-align: center;">
<span style="font-family: Arial,Helvetica,sans-serif;">4367:9987:a01b:0000:0000:0007:cafe:babe</span></div>
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">According to the simplification rules of IPv6 addresses, this can be written as</span><br />
<br />
<div style="text-align: center;">
<span style="font-family: Arial,Helvetica,sans-serif;">4367:9987:a01b::7:cafe:babe</span></div>
<div style="text-align: center;">
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
</div>
<div style="text-align: left;">
<span style="font-family: Arial,Helvetica,sans-serif;">Since with a double colon you can jump through all of the all-zero address parts, and leading zeros can also be omitted.</span><br />
<br />
<a name='more'></a><br />
</div>
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
What will the Romans ever
do for us?</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">We did not only sucked almost all the oil out of the Earth's crust, but we <i>really</i> almost used up all of the IPv4 addresses.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Now look around you and take a mental note of the feeling of right now, because when your grandchildren will ask you how it felt when all of the IPv4 addresses were used up, you will have to describe this exact feeling.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Our oil supplies will be enough for quite a few decades, but out IPv4 addresses won't last an other one. Our pool of IPv4 addresses is quickly running dry.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">IPv6 offers us a mindboggling 2<sup>128</sup> addresses. That's right, since the original IPv4 address pool size were a mere 2<sup>64</sup>, the new pool size will be the old pool size <i>squared</i>, so every device that has an IPv4 address today could get as many IPv6 addresses as there are IPv4 addresses on the face of the Earth.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">And that is a lot. </span><span style="font-family: Arial,Helvetica,sans-serif;">(Not counting with NAT of course ;) )</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">If the IETF would make this happen again (<i>squaring</i> the address pool), then a new IP address could be assigned to every thousandth or so atom <i>in the known universe</i>.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Besides the exuberant addresses, IPv6 brings to us an other few things play with, such as:</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<br />
<ul>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Direct connectivity, no NATs. Again, for there are a plenty of available addresses.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Nice zero-configuration LANs, no router or other coordination is needed.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Multicast - send traffic to many hosts at once.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Anycast - send traffic to the closest host holding the given anycast address.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Simpler message format, faster routing, faster Internet.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Built-in security</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Better QoS - better handling and controlling of independent data streams between hosts, making possible to throttle and prioritize traffic more efficiently.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Better mobility support - switching between networks is much easier.</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Better administration: zero configuration, network renumbering and easy <i>multihoming</i> (using more than one internet connections).</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;">Smooth transition from IPv4</span></li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR02lnX05rZpOyiM5bx2TAoh_hd9lIQ1WiQIG1Hww_gt-unQoPX9VvBgWtVtk4chh_vQHgycTv3_ZY3RqFNvbDNzFWbeJO9kNiCx6KVxVUe8wW65ZK00VvfyvRY183JRz_4csH_ISopDs/s1600/heavy_breathing.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR02lnX05rZpOyiM5bx2TAoh_hd9lIQ1WiQIG1Hww_gt-unQoPX9VvBgWtVtk4chh_vQHgycTv3_ZY3RqFNvbDNzFWbeJO9kNiCx6KVxVUe8wW65ZK00VvfyvRY183JRz_4csH_ISopDs/s1600/heavy_breathing.jpg" height="295" width="320" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">This last bullet point made possible by being able to "map" all the current IPv4 addresses into a tiny little fraction of the available IPv6 addresses.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">This with the availability of IPv6 <i>tunnel</i> services (like SIXXS) can ship you into the future without causing severe (or any) seasickness.</span><br />
<br />
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
SIXXS?</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;"><i>SIXXS</i> is an IPv6 deployment & tunnel broker. Anyone can register and get an IPv6 address <i>range</i> the size of the current IPv4 address space. I for example own three of them at the moment.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">If you register at https://www.sixxs.net/ you can get an IPv6 address range and a possibility to build a <i>tunnel</i> through the <i>old Internet</i> into the <i>new one</i>.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br />Tunnelling works by directing all non-local IPv6 traffic to a distant machine, with IPv6 packets encapsulated in IPv4 ones. You might not get the speed benefits, but you will get all the others, with the priceless feeling of being an early adopter of a wonderful new technology.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">You can then browse the Web and use almost all your usual applications over IPv6, and can access the growing number of resources only available on the new net - on hosts with <i>only</i> IPv6 addresses.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Your SIXXS account will be credited with a few <i>ISK</i>s when you open it. A handful of ISKs are need to request your tunnel (and therefore address (range)). ISKs accumulate by keeping your tunnels up and running - something you probably planning to do anyway.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">If you would like to experiment with your home devices, you need an IPv6 - and SIXXS tunnel - capable router.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Bummer.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Not many routers can set up tunnels to SIXXS <i>PoP</i>s out of the box, so we need to do some geeky stuff to convert your household to IPv6.</span><br />
<br />
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
Raspberry PI?</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">You probably know - and chances you own - a small <i>single board computer</i> called a Raspberry PI. This little device has a slow-but-OK ARM CPU, some OK amount of memory for a very low price. If you don't have one, make sure you buy two, for about a hundred USDs it's a fair deal.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Google for your local supply of PIs. </span><br />
<br />
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
Raspberry PI setup - aiccu</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">Raspberry PI can run Linux, make sure you have the current official version of <i>Raspbian</i> running on it.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">After you have requested your tunnel - and it got <i>approved</i> in a few hours (or days if you are unlucky), you can set up your PI to connect to the new Internet.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Make sure you request a <i>dynamic, NAT traversing AYIYA tunnel</i>, since it is almos certain that you're behind a NAT.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The client program you need to install is <i>aiccu</i>. Make that happen by issuing the following command:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">sudo apt-get install aiccu</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">This will install the software and the dependencies. The aiccu install script will ask for your user name and password, and if you have more than one tunnel, you will have to select the appropriate one here.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Aiccu will start up, and there you go:</span><br />
<br />
<pre>root@minotaur:~# ping6 google.com
PING google.com(bud02s04-in-x06.1e100.net) 56 data bytes
64 bytes from bud02s04-in-x06.1e100.net: icmp_seq=1 ttl=57 time=4.99 ms
64 bytes from bud02s04-in-x06.1e100.net: icmp_seq=2 ttl=57 time=7.76 ms
64 bytes from bud02s04-in-x06.1e100.net: icmp_seq=3 ttl=57 time=6.24 ms
64 bytes from bud02s04-in-x06.1e100.net: icmp_seq=4 ttl=57 time=3.86 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 3.865/5.719/7.767/1.452 ms
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">You can see into the future. Minotaur is my Raspberry PI, the command I used is ping6, the target machine was google.com.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">But what about the other devices on the network? They can't. Your RPI has to act as a router between them and the rest of the IPv6 Internet.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Fortunately this can easily be done. Just install <i>radvd</i>, a router advertisement daemon that will periodically yell at your local devices, telling them about the great opportunity of becoming a test subject for your experiment:</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;">sudo apt-get install radvd</span></span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Arial,Helvetica,sans-serif;">Radvd</span> </span>needs a configuration file at /etc/radvd.conf:</span><br />
<br />
<pre>interface eth0 {
AdvSendAdvert on;
prefix 2a01:368:e000:8074::/64 {
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
};
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Now hold on for a while, <b>that is my ip address prefix</b>, you have to substitute that for your own. Take a look at your SIXXS User Home page under the title "Subnets", and look for "Subnet Prefix".</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">If you start radvd after this, well, it might not want to start, complaining about IPv6 forwarding being turned off.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
Raspberry PI setup - sysctl</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">Allow IPv6 forwarding by creating the configuration file /etc/sysctl.d/local.conf containing:</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<br />
<pre>net.ipv6.conf.all.forwarding=1</pre>
<br />
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
Raspberry PI setup - interfaces</span></h2>
Linux won't forward IPv6 packets from an interface if there is no IPv6 address of that interface is known. So we will have to set one up.<br />
<br />
Edit /etc/network/interfaces and add<br />
<br />
<pre>iface eth0 inet6 static
pre-up modprobe ipv6
address 2a01:368:e000:8074::101
netmask 64
</pre>
<br />
to it. Please <b>replace the IPv6 address</b> to one with your SIXXS-provided prefix. You can make one up by writing a suitable number after your prefix.<br />
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">Raspberry PI setup - modules </span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">The sysctl part only works if the ipv6 kernel module is loaded before sysctl tries to set up forwarding. This can be done by editing the file /etc/modules and inserting a single line into it:</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">ipv6</span> </span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">This way the proper kernel driver will be loaded at every boot. </span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">After all of this, you need to reboot your PI. Make sure aiccu and radvd is running.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<br />
<h2>
<span style="font-family: Arial,Helvetica,sans-serif;">
Set up your other devices.</span></h2>
<span style="font-family: Arial,Helvetica,sans-serif;">Hah! I was joking. You don't have to. Actually I had to reconnect to my wifi with my Samsung Galaxy S4 to notice the change, but my desktop computer picked up the router configuration without me having to intervene.</span><br />
<br />
<div style="text-align: center;">
<span style="font-family: Arial,Helvetica,sans-serif;">The IPV6 test site <a href="http://test-ipv6.com/%20%20">http://test-ipv6.com/</a> scored 10/10.</span></div>
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-70337718547554851652013-10-26T07:00:00.002-07:002013-10-26T08:08:02.519-07:00Amateur (HAM) radio: analyzing historical data from reversebeacon.net<h3>
What is amateur radio?</h3>
Amateur radio (or HAM radio) is a great hobby, or sport - it depends on how seriously do you take it, and what activities you choose to engage in precisely.<br />
<br />
Basically HAM radio is a pastime where people around the world buy and/or build radio transceivers, antennas and lots of other gear, and make contacts with each other using the frequencies and modes that complies with rather strict regulations.<br />
<br />
Amateur radio operators also participate in emergency communication, since no infrastructure is required, and with the proper tools, they can talk over thousands of kilometres.<br />
<br />
Despite the aforementioned strictness of radio regulations, there are multitude of ways you can "do HAM radio".<br />
<br />
If you enjoy chatting, you can buy a cheap hand-held device and check into a "net" on a local repeater. If you like "hunting DX" - making long-distance contacts on different continents (or covering an unusually large distance at the given circumstances), than you might want to invest into a good short wave radio and a suitable antenna.<br />
<br />
You can also build your own radio, antenna and necessary equipment AND hunt DX with that!<br />
<br />
<h3>
What is reverse beacon?</h3>
Reverse beacon (http://reversebeacon.net/) is a service that gathers information from hundreds of radio receivers around the world. These radio receivers listen on various bands, constantly trying to decode the communication.<br />
<br />
The network currently mostly listens to "CW" - continuous wave communication or other words Morse code.<br />
<br />
Yepp, Morse code is still in use, actually this method is very popular in the amateur radio community. It is particularly easy to build Morse receivers and transmitters, so a lot of the DIY folks speak Morse.<br />
<br />
<a name='more'></a>Also, Morse code can be decoded automatically, so reverse beacon can intercept amateur radio Morse code communication.<br />
<br />
The intercepted communication can be searched and displayed on the website, even in real time. Cool.<br />
<br />
A very nice feature of the RBN is that it makes the historical data available for us. Free. Thank you folks!<br />
<br />
RBN stores various interesting info about the intercepted messages. Who sent it, who heard it and when; how loud was the signal and how fast the pace of the dits and dahs were.<br />
<br />
Today I've counted 221.582.054 rows in the database. That's a big pile of data to chew on. Let me tell you how I do it.<br />
<br />
<h3>
Choosing a database engine</h3>
Considering the size of the data set, you may call this "big data". You could shove it into mysql or postgresql, or something similar, but you'd need to build proper indexes, or just need to be very patient. Loading the data also might take long, hours maybe.<br />
<br />
My choice is <a href="http://infinidb.org/">InfiniDB</a>.<br />
<br />
InfiniDB is a <i>columnar</i> database. There is a free, and an enterprise version. The free is more than fine for us.<br />
<br />
It stores in a column continuous-, instead of row continuous fashion. This (and many other design features) is to greatly reduce IO: reading 220 <i>million</i> rows from the disk is no joke in case you have to do a full table scan. InfiniDB <i>never</i> do that.<br />
<br />
Instead it does a <i>full column scan</i>, so it won't read data you don't need: in mysql, a simple select <column> from <table> would read or skip through all data in the table, including columns you don't need (well, this is more complicated, but for now this is enough).<br />
<br />
You also don't need to create indexes. Actually, the data is structured so that you could say that in InfiniDB <i>data is the index</i>. Whatever query you throw on InfiniDB, it will be kinda fast. (See the <a href="http://www.calpont.com/phocadownload/documentation/2.0.0/CalpontInfiniDBConceptsGuide_20-1.pdf">concept guide</a> for details. Also, you might want to read a bit about <a href="http://infinidb.org/dmdocuments/Row%20Database%20vs%20InfiniDB%20Benchmark%20White%20Paper.pdf">how InfiniDB compares to other, row-based databases</a>.)<br />
<br />
Unless you want to do some crazy SQL magic, which just won't work, or will be very slow, since InfiniDB does not support SQL entirely, <a href="http://www.calpont.com/phocadownload/documentation/2.0.3/CalpontInfiniDBSyntaxGuide_203-1.pdf">only a subset</a> of it.<br />
<br />
<h3>
Installing InfiniDB</h3>
You can get the installer after registering and logging into infinidb.org.<br />
<br />
The installation procedure is simple, read the <a href="http://infinidb.org/resources/getting-started">getting started guide</a>.<br />
<br />
Basically what you have to do is to uncompress the downloaded archive under /usr/local. It will create a directory /usr/local/Calpont. After that, run /usr/local/Calpont/bin/install-infinidb.sh as root.<br />
<br />
The InfiniDB isntaller is made for servers, so you have to adjust certain parameters to be able to run InfiniDB correctly.<br />
<br />
Open up /usr/local/Calpont/etc/Calpont.xml, and look for a line like this:<br />
<br />
<NumBlocksPct>66</NumBlocksPct><br />
<br />
The number between the XML tags is a percentage. InfiniDB will acquire this percentage of the totaly physical system RAM for it's own block cache. I've seen this value set by the installer as high as 80%, and it's almost always unacceptable on a desktop computer. Change this to whatever you think is OK.<br />
<br />
Also, please try to set this as high as you can easily tolerate, since the block cache can (and will) make queries faster by reducing, or eliminating disk IO.<br />
<br />
Fortunately I own a computer with 32 gigabytes of RAM, and the total InfiniDB data directory is about 27 gigabyte, so I can store most of the reverse beacon data in my RAM. After a few queries, things speed up quite a bit, and further queries don't even make my HDD LED blink.<br />
<br />
Since we will use a single table for the RBN data (wich is kinda suboptimal, but hey), we don't really need to care about and optimize join behaviour, but InfiniDB tuning is quite an interesting topic, so if you too care, read the <a href="http://www.calpont.com/phocadownload/documentation/2.0.3/CalpontInfiniDBTuningGuide_203-1.pdf">tuning guide</a>.<br />
<br />
<h3>
Getting the historical data</h3>
First of all, InfiniDB is "just" a mysql storage engine, so by installing inifidb, you install a mysql instance as well. Make sure to stop any running mysql databases before attempt to install infinidb. (Also, you can configure either InfiniDB or your existing mysql to run on a different than the default 3306 port, by editing the proper my.cnf file.)<br />
<br />
You can download raw data from <a href="http://www.reversebeacon.net/raw_data/">http://www.reversebeacon.net/raw_data/</a>. It not so entertaining to download and process almost two thousand files by hand, so I've written a python script to do the job. Please be polite, and don't comment the out the line that waits between downloads - I don't want anyone to slam the server.<br />
<pre class="brush: python">#!/usr/bin/env python
# -*- coding: utf8 -*-
import os
import time
import urllib2
import datetime
base_url = "http://www.reversebeacon.net/raw_data/dl.php?f="
today = datetime.date.today()
delta = datetime.timedelta(1)
i = datetime.date(2009, 2, 21)
while i <= today:
datestr = "%d%02d%02d" % (i.year, i.month, i.day)
fname = "RBNDATA/%s.zip" % datestr
url = base_url + datestr
if os.path.isfile(fname):
print "%s exists." % fname
i += delta
continue
print "Downloading %s..." % url
try:
content = urllib2.urlopen(url)
f = open(fname, "w")
f.write(content.read())
f.close()
except urllib2.HTTPError:
print "Error downloading %s" % url
i += delta
time.sleep(5)
</pre>
<br />
Put this script into an empty directory as "download_rbndata.py" and also create a directory named "RBNDATA" next to it. Cd into the script's directory, and run it with python or pypy (issue "python download_rbndata.py", or "pypy download_rbndata.py" command, or just give it an executable flag and fire ./download_rbndata.py).<br />
<br />
The download will take about 3 hours. Be patient. Once you've downloaded the files, you can fire the command again, and it will not download the files you already have, so you can gradually gather your own pile of RBN data without re-downloading everything over and over.<br />
<br />
If you set up InfiniDB properly, you need one more trick to get a DB shell.<br />
<br />
Open up a bash shell and do<br />
<br />
. /usr/local/Calpont/bin/calpontAlias<br />
<br />
Notice the space between the dot and the rest of the line!<br />
<br />
This script will set up command aliases, so - for example - you can get an infinidb shell, by issuing:<br />
<br />
idbmysql<br />
<br />
After this, a familiar mysql shell will greet you. If you don't want to type the calpontAlias command every time you open a shell, make sure you include it in your .bashrc file. (This step is optional)<br />
<br />
Also, you might want to put /usr/local/Calpont/bin into your PATH as well. (This is not vital either.)<br />
<br />
<h3>
Converting and improving on quality</h3>
We will use InfiniDB's cpimport tool to import the data into the database.<br />
<br />
But first we have to take care of a few problems:<br />
<br />
The format of the files are slightly different in different time intervals. Some have headers, some don't, there are files with more columns than others, and the last row contains no data.<br />
<br />
Also, cpimport can't read ZIPs, so we have to uncompress the whole thing and make it into a single csv file.<br />
<br />
All these things are taken care of by the following python script. Save this as "build_import.py", and put next to the file and RBNDATA directory described above.<br />
<pre class="brush: python">#!/usr/bin/env python
# -*- coding: utf8 -*-
import os
import glob
import zipfile
basedir = "RBNDATA"
importfile = "rbndata.csv"
zips = glob.glob(basedir + "/*.zip")
zips.sort()
impf = open(importfile, "w")
for z in zips:
f = zipfile.ZipFile(z, "r")
content = None
csvname = f.namelist()[0]
print csvname
csv = f.open(csvname, "r")
for line in csv:
if line[:8] == 'callsign':
print "(dropping header)"
continue
if line[:1] == '(':
print "(dropping last line)"
continue
extra = 12 - line.count(',')
impf.write(line.rstrip() + (',')*extra + "\n")
csv.close()
f.close()
impf.close()
</pre>
<br />
Run this just like the other script. It will read the ZIP files and create a <b>huge</b> csv (more than 16 gigabytes). This can be imported into InfiniDB.<br />
<br />
<h3>
Importing the data </h3>
Make sure your InfiniDB instance is up and running (use /etc/init.d/infinidb start if it isn't).<br />
<br />
Start an idbmysql shell, and run the following sql:<br />
<br />
<pre class="brush: sql">CREATE DATABASE rbndata;
use rbndata;
CREATE TABLE rbndata (
callsign VARCHAR(16),
de_pfx VARCHAR(6),
de_cont VARCHAR(4),
freq DECIMAL(12,4),
band VARCHAR(6),
dx VARCHAR(16),
dx_pfx VARCHAR(6),
dx_cont VARCHAR(4),
mode VARCHAR(6),
db INT,
logged_at DATETIME,
speed INT,
tx_mode VARCHAR(6)
) ENGINE=InfiniDB DEFAULT CHARSET=latin1;
</pre>
<br />
<br />
Now you can use cpimport to do the job:<br />
<br />
$ sudo /usr/local/Calpont/bin/cpimport rbndata rbndata rbndata.csv -s ,
<br />
<br />
Of course, you have to stand in the directory where the <i>huge</i> rbndata.csv file is, which was created by out python scripts.<br />
<br />
Also notice the comma after the -s switch. This is how we set the field separator, so it's absolute mandatory. <br />
<br />
Also, as the saying goes, "this might take a few minutes". Indeed, it only took a few, my computer chugged this bottle of bytes down in an amazing 510.5 seconds, swallowing about <b>430 thousand rows per second</b>. Wow.<br />
<br />
You can verify the bulk load by issuing the following command at the idbmysql shell:<br />
<pre>mysql> select count(*) from rbndata;
+-----------+
| count(*) |
+-----------+
| 221582054 |
+-----------+
1 row in set (2.42 sec)
</pre>
<br />
This might be a lot slower in your case, since at this time, your block cache is cold, and you have to read a lot from the disk at this time. My cache was hot, and InfiniDB only had to hit RAM to calculate the rows in the table.<br />
<br />
Your results also may vary, since after the publication of this article many more contacts were logged by the RBN.<br />
<br />
<h3>
</h3>
<h3>
Some interesting queries </h3>
Now we have an InfiniDB up and running, have our data inside it, so let's the fun begin!<br />
<br />
We already see how we calculate the total number of rows, which is simple and boring.<br />
<br />
Let's see the different callsigns!<br />
<br />
select count(distinct dx) from rbndata;<br />
<br />
There is an astonishing 690,843 different callsigns in the database. Of course a callsign might be there as a (let's call it) a "simple" one, like mine: HA5FTL, or the operator might have worked outside the shack, on a field, so HA5FTL/P -like callsigns are also there, so a single station or operator might even appear 3-4 different way. <br />
<br />
Counting just the simple cases gives us:<br />
<br />
select count(distinct dx) from rbndata where dx not like '%/%';<br />
<br />
614,185 different callsigns.<br />
<br />
Let see portables: <br />
<br />
select count(distinct dx) from rbndata where dx like '%/P';<br />
16,409<br />
<br />
Mobiles:<br />
select count(distinct dx) from rbndata where dx like '%/M';<br />
4783<br />
<br />
Maritime mobiles:<br />
select count(distinct dx) from rbndata where dx like '%/MM';<br />
873<br />
<br />
Air mobiles:<br />
select count(distinct dx) from rbndata where dx like '%/AM';<br />
34<br />
<br />
Ok, so how much automatic receiver stations are (were) in the RBN?<br />
select count(distinct callsign) from rbndata;<br />
895 is the result. That's a lot of stations!<br />
<br />
Since listening stations come and go, let's see how many were working in august:<br />
select count(distinct callsign) from rbndata where logged_at between '2013-08-01' and '2013-09-01';<br />
Result is 197. Notice the speed: this query run <i>under one second</i> on my machine.<br />
<br />
Cool isn't it? And it gets way cooler than that.<br />
<br />
See how much loggers on each continent:<br />
<br />
select de_cont, count(distinct callsign) as loggers from rbndata group by de_cont order by loggers desc;<br />
<br />
And the result is:<br />
<ul>
<li>EU: 502</li>
<li>NA: 284</li>
<li>AS: 53</li>
<li>SA: 33</li>
<li>OC: 19</li>
<li>AF: 5</li>
</ul>
So Europe has the largest number of logger stations, and Africa has disappointingly few. Also, Asia and South America could use some volunteers, and the people of Oceania are also have to pull themselves together ;)<br />
<br />
You can do the same with the DXes, or the call signs that were recoded:<br />
<br />
select dx_cont, count(distinct dx) as dxes from rbndata group by dx_cont order by dxes desc;<br />
<br />
<ul>
<li>EU: 376185</li>
<li>NA: 178105</li>
<li>AS: 83266</li>
<li>AF: 19333</li>
<li>SA: 17755</li>
<li>OC: 16859</li>
</ul>
No surprise, EU leads the list, North America follows. But what's interesting is Africa's position. This is probably because Africa is a popular target of DX expeditions: trips to places that rarely see HAM radio activity due to the lack local amateurs, or even human beings (like small bare-rock islands in the middle of nowhere). Africa also have a HAM radio life of it's own, in contrast to what you might think first.<br />
<br />
(Hm, I miss Antarctica. Don't laugh, there IS some HAM radio activity there, I even made contact with RI1ANF, a station at Bellinghausen Base, King George Island, which technically belongs to Antarctica).<br />
<br />
We can play with the bands too. Let's see what bands are the most popular (or at least the most popular in RBN's database):<br />
<br />
select band, count(distinct dx) as c from rbndata group by band order by c;<br />
<br />
<ul>
<li>20m: 378063</li>
<li>40m: 372545</li>
<li>80m: 203150</li>
<li>15m: 173935</li>
<li>30m: 120276</li>
<li>10m: 93178</li>
<li>17m: 82932</li>
<li>160m: 63508</li>
<li>12m: 36151</li>
<li>6m: 25389</li>
<li>2m: 2595</li>
<li>60m: 1200</li>
<li>4m: 162</li>
<li>472kHz: 112</li>
<li>70cm: 66</li>
<li>137kHz: 1</li>
</ul>
Aaaand the great trio leads: indeed, 20, 40 and 80 meters are very popular, and you can always find <i>something</i> on 30 meters, and when the propagation is good, great DXes keep popping up on 15 meters.<br />
<br />
If you lack antenna space, I'd definitely recommend you to build <i>something</i> (a wire antenna, or an aluminium stick) that does not too bad at 15 and 20 meters, and you just <i>might</i> be able to work with local stations on 40 meters up to maybe 1-2 thousand kilometres, if you're lucky (and that stations has a proper antenna).<br />
<br />
An other good question is: how each of these band perform at great distances? unfortunately I don't have the exact location of these stations, however I could buy access to qrz.com's huge database, and grab location information there.<br />
<br />
Let's define a "DX log entry" as an entry that have different data in the dx_cont and de_cont field, e.g. the logger station and the calling stations were on a different continent:<br />
<br />
select band, count(*) as c from rbndata where de_cont <> dx_cont group by band order by c desc;<br />
<br />
This will give us the following list:<br />
<br />
<ul>
<li>20m: 25547420</li>
<li>40m: 15628909</li>
<li>15m: 12397729</li>
<li>10m: 4076702</li>
<li>80m: 3660948</li>
<li>17m: 2848269</li>
<li>30m: 2701406</li>
<li>12m: 1012281</li>
<li>160m: 927274</li>
<li>6m: 42104</li>
<li>60m: 5734</li>
<li>2m: 895</li>
<li>4m: 119</li>
<li>472kHz: 10</li>
<li>70cm: 4</li>
</ul>
So if you want to work DX, you will probably have luck on 20 meters (also don't forget that the data is skewd, since the distribution of logger stations is not uniform).<br />
<br />
We can also ask the question: which band is more like a "DX band", and wich is more of a "local band"? Let's see:<br />
<br />
select band, sum(if(de_cont <> dx_cont,1,0))/count(*) as c from rbndata group by band order by c desc;<br />
<ul>
<li>12m: 0.6003</li>
<li>10m: 0.5405</li>
<li>15m: 0.5262</li>
<li>17m: 0.4434</li>
<li>20m: 0.3697</li>
<li>40m: 0.2380</li>
<li>30m: 0.2296</li>
<li>60m: 0.1714</li>
<li>80m: 0.1433</li>
<li>160m: 0.1001</li>
<li>4m: 0.0474</li>
<li>6m: 0.0445</li>
<li>2m: 0.0238</li>
<li>70cm: 0.0071</li>
<li>472kHz: 0.0036</li>
<li>137kHz: 0.0000</li>
</ul>
So what we've got here? The ration of DX/same continent contacts is the higher on 12 meters. We can speculate that you'll do more DX on 12 meters than not. (Of course, never forget that the data is heavily biased. We should examine the distribution of listening stations through bands and continents and apply some clever de-biasing before ever stating something like that!)<br />
<br />
Despite the bias, the figure above are somewhat reflect reality: the higher HF bands are more like DX bands, and the lowers are generally easier to use for local communications. This is because the middle (and sometimes the upper) HF frequencies are more likely to be reflected from the upper atmosphere. The low frequencies are absorbed, the high frequencies are leaving the Earth into space. (Yes I know, this is <i>much</i> more complicated, and the reflectivity of the ionosphere is a science of it's own. Also it's noteworthy that it is a very cool piece of science that potentially can greatly benefit from the data gathered by the RBN.)<br />
<br />
I have a humble station with an Elecraft K2 as my main radio along with a few work-in-progress DIY rigs. My antenna is also just a 4 metres long stick of aluminium tube sticking out from my window.<br />
<br />
How such <i>amateur</i> station can perform?<br />
<br />
Let's see:<br />
<br />
select dx, count(*) from rbndata where dx LIKE 'HA5FTL%' group by dx;<br />
<br />
<ul>
<li>HA5FTL: 716</li>
<li>HA5FTL/P: 32</li>
</ul>
Well, that's not much. Partly because of my low-level signal, partly because I rarely "call CQ", so my callsign is mostly only heard when I answer to a general call. Weak CQ-s are rarely answered, so I listen and answer calls myself mostly.<br />
<br />
But how the continents can hear me?<br />
<br />
select dx, de_cont, count(*) as c from rbndata where dx like 'HA5FTL%' group by dx, de_cont order by dx, c desc;<br />
<br />
<pre>+----------+---------+-----+
| dx | de_cont | c |
+----------+---------+-----+
| HA5FTL | EU | 654 |
| HA5FTL | NA | 46 |
| HA5FTL | AS | 14 |
| HA5FTL | AF | 1 |
| HA5FTL | SA | 1 |
| HA5FTL/P | EU | 31 |
| HA5FTL/P | AS | 1 |
+----------+---------+-----+
7 rows in set (6.99 sec)
</pre>
<br />
So you only have a good chance to hear me if you're living in Europe. Also if you own a decent antenna, we might be able to make contact if you're living in Canada's or the USA's east coast. Asian and African HAMs will have a hard time hear me at all. (Also, this low number of logs can be attributed to the low number of loggers at AF and AS, but let's just be realistic: my shack is <i>far</i> from ideal).<br />
<br />
Let's see my favourite bands (beside Infected Mushroom and Shpongle :P):<br />
<br />
select dx, band, count(*) as c from rbndata where dx like 'HA5FTL%' group by dx, band order by dx, c desc;<br />
<br />
<pre>+----------+------+-----+
| dx | band | c |
+----------+------+-----+
| HA5FTL | 40m | 288 |
| HA5FTL | 20m | 255 |
| HA5FTL | 15m | 100 |
| HA5FTL | 17m | 31 |
| HA5FTL | 80m | 23 |
| HA5FTL | 10m | 10 |
| HA5FTL | 30m | 9 |
| HA5FTL/P | 40m | 26 |
| HA5FTL/P | 10m | 3 |
| HA5FTL/P | 20m | 3 |
+----------+------+-----+
10 rows in set (7.04 sec)
</pre>
<br />
Indeed, I mostly work on 40 and 20, occasionally on 15 meters, rarely on others.<br />
<br />
We also could see the "DX ratio" on each band:<br />
<br />
select dx, band, sum(if(dx_cont <> de_cont,1,0))/count(*) dx_ratio, count(*) calls_heared from rbndata where dx like 'HA5FTL%' group by dx, band order by dx, dx_ratio desc;<br />
<br />
I've included the calls_heared column so we can see how precise the dx_ratio might be:<br />
<br />
<pre>+----------+------+----------+--------------+
| dx | band | dx_ratio | calls_heared |
+----------+------+----------+--------------+
| HA5FTL | 15m | 0.2200 | 100 |
| HA5FTL | 10m | 0.2000 | 10 |
| HA5FTL | 20m | 0.1216 | 255 |
| HA5FTL | 17m | 0.0968 | 31 |
| HA5FTL | 40m | 0.0139 | 288 |
| HA5FTL | 80m | 0.0000 | 23 |
| HA5FTL | 30m | 0.0000 | 9 |
| HA5FTL/P | 20m | 0.3333 | 3 |
| HA5FTL/P | 40m | 0.0000 | 26 |
| HA5FTL/P | 10m | 0.0000 | 3 |
+----------+------+----------+--------------+
</pre>
<br />
It looks like it's worth getting on 15 meters if I want to work DX from home, and If the /P 20 meters dataset size would be bigger at the same ratio, I'd say that 20 meters with a random wire on a field is good for DX, unfortunatel 3 logged contacts is waaay too low to mean anything.<br />
<br />
I must note this point that all of these queries run under half a minute, and mostly under 10 seconds. On 200 million rows with group by-s and stuff. <br />
<br />
Well I think you all get the picture now. If you have some interesting queries, or even questions, just write a comment below.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-90737614367925746082013-05-13T08:02:00.003-07:002013-12-23T22:17:40.255-08:00Programming the Bluegiga BLE112 Bluetooth 4.0 module with Linux<h2>
Why? </h2>
BLE112 is a Blutooth 4.0 or Blutooth Low Energy (BLE for short) module that contains a microcontroller packed with an awsome firmware that lets you write Bluetooth 4.0 applications in a very user-friendly language called BGScript.<br />
<br />
<a name='more'></a><br />
<b>WARNING!!! if you follow the guide below, you're gonna brick the chip. The recent versions of BLE* firmwares require a license key that built into the chip, and cc-tool wipes it, effectively disabling the radio on the device. The solution is to use the "BLE Update Utility" from BlueGiga. I couldn't start it on older Windows installations nor on Wine. Work in progress. If you manage to start it in Wine, please share the knowledge. Thank you!</b> <br />
<br />
You can get more info on the module at:<br />
<br />
<a href="http://www.bluegiga.com/BLE112_Bluetooth_Smart_module">http://www.bluegiga.com/BLE112_Bluetooth_Smart_module</a> <br />
<br />
If you want to use the module, you need a programmer tool from Texas Instruments called cc-debugger:<br />
<a href="http://www.blogger.com/goog_1386200563"><br /></a>
<a href="http://www.ti.com/tool/cc-debugger">http://www.ti.com/tool/cc-debugger</a><br />
<br />
And you need to wire up the chip to the computer. You can read about that at the following two links:<br />
<br />
<a href="http://blog.bluetooth-smart.com/2012/09/11/programming-the-ble112-with-c-code-using-iar/">http://blog.bluetooth-smart.com/2012/09/11/programming-the-ble112-with-c-code-using-iar/</a><br />
<br />
<a href="http://blog.bluetooth-smart.com/2012/09/16/programming-the-ble112-using-bgscript/">http://blog.bluetooth-smart.com/2012/09/16/programming-the-ble112-using-bgscript/</a><br />
<br />
The programming software can be downloaded from bluegiga's techforum. These are a collection of windows applications that let you build the program and there's also a nice tool called blegui2.exe that lets you debug you solution.<br />
<br />
To try your bluetooth module, you need a bluetooth 4.0 enabled hardware or a bluetooth 4.0 dongle such as BLED112 from Bluegiga:<br />
<br />
<a href="http://www.bluegiga.com/BLED112_Bluetooth_smart_dongle">http://www.bluegiga.com/BLED112_Bluetooth_smart_dongle</a><br />
<br />
This plugs into an USB port. There is a proper driver that works with Windows. I had no problem to set up the software and hardware on my home machine that runs Windows 8. When I switched to Linux, problems popped up.<br />
<br />
First of all, I noticed that when I plugged in the dongle, it kept disconnecting, and connecting again. The following output is from dmesg:<br />
<br />
<pre>[ 5755.444220] cdc_acm 3-1:1.0: ttyACM3: USB ACM device
[ 5755.984120] usb 3-1: USB disconnect, device number 43
[ 5756.524111] usb 3-1: new full-speed USB device number 44 using uhci_hcd
[ 5756.698098] usb 3-1: New USB device found, idVendor=2458, idProduct=0001
[ 5756.698102] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5756.698104] usb 3-1: Product: Low Energy Dongle
[ 5756.698106] usb 3-1: Manufacturer: Bluegiga
[ 5756.698108] usb 3-1: SerialNumber: 1
[ 5756.706137] cdc_acm 3-1:1.0: ttyACM3: USB ACM device
[ 5757.224143] usb 3-1: USB disconnect, device number 44
[ 5757.744071] usb 3-1: new full-speed USB device number 45 using uhci_hcd
[ 5757.912164] usb 3-1: New USB device found, idVendor=2458, idProduct=0001
[ 5757.912175] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5757.912182] usb 3-1: Product: Low Energy Dongle
[ 5757.912188] usb 3-1: Manufacturer: Bluegiga
[ 5757.912194] usb 3-1: SerialNumber: 1
[ 5757.920274] cdc_acm 3-1:1.0: ttyACM3: USB ACM device
[ 5758.464106] usb 3-1: USB disconnect, device number 45
[ 5758.988078] usb 3-1: new full-speed USB device number 46 using uhci_hcd
[ 5759.160089] usb 3-1: New USB device found, idVendor=2458, idProduct=0001
[ 5759.160100] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5759.160107] usb 3-1: Product: Low Energy Dongle
[ 5759.160113] usb 3-1: Manufacturer: Bluegiga
[ 5759.160119] usb 3-1: SerialNumber: 1
[ 5759.168192] cdc_acm 3-1:1.0: ttyACM3: USB ACM device
[ 5759.704175] usb 3-1: USB disconnect, device number 46
[ 5760.220064] usb 3-1: new full-speed USB device number 47 using uhci_hcd
[ 5760.592121] usb 3-1: New USB device found, idVendor=2458, idProduct=0001
[ 5760.592126] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5760.592128] usb 3-1: Product: Low Energy Dongle
[ 5760.592130] usb 3-1: Manufacturer: Bluegiga
[ 5760.592132] usb 3-1: SerialNumber: 1</pre>
<br />
I dug through the Internet for a good few days after I found out that this dongle resets itself if it receives a malformed command. I suspected that there is some process that tries to communicate with it and keeps it resetting over and over.<br />
<br />
The solution was to add the device to the udev blacklist. Create a new file under /lib/udev/rules.d (mine is named 77-mm-my.rules), and add the following content:<br />
<br />
<pre># BLuegiga BLED112
ATTRS{idVendor}=="2458", ATTRS{idProduct}=="0001", ENV{ID_MM_DEVICE_IGNORE}="1"
</pre>
<br />
The name of the file is important. Use my idea about the name if you don't know udev very well.<br />
<br />
After this, restart modem-manager by issuing "sudo killall modem-manager". After this, modem-manager should not interfere with the device.<br />
<br />
This trick can save a lot of hardware from being bugged to death by modem-manager. One example is my USB AVR programmer.<br />
<br />
After this, ALL bluegiga programs kinda work as expected with wine. No freezes or segfaults! The compliler and builder can be run exactly as you would do from under Windows. Just pay attention to the path / drive mapping where appropriate.<br />
<br />
The big surprise is that blegui2.exe is also works like a charm.<br />
<br />
The only problem is with wine itself. It does not seem to offer out-of-the-box serial port support.<br />
<br />
I managed to find a solution to this problem too:<br />
<br />
<a href="http://ubuntuforums.org/showthread.php?t=1523814">http://ubuntuforums.org/showthread.php?t=1523814</a><br />
<br />
TL;DR: you have to download a registry file, and integrate it to wine's registry: <br />
<br />
$ wget -O comport.reg http://bugs2.winehq.org/attachment.cgi?id=10210<br />
$ regedit comport.reg<br />
<br />
After this, wine can show Linux serial devices a Windows comX ports.<br />
<br />
To configure the mapping, you have to create symlinks under ~/.wine/dosdevices/comX which point to devices under /dev/ttyXXX. In my case:<br />
<br />
<pre>$ cd ~/.wine/dosdevices
$ ln -s /dev/ttyACM3 com1
</pre>
<br />
It's actually quite a dull process to look up the device in dmesg and create the symlink myself. Udev can do this! Let's modify the udev rule to create a device symlink to our dongle, and point wine's com1 symlink to that symlink (yo dawg... :) ):<br />
<br />
<pre># BLuegiga BLED112
ATTRS{idVendor}=="2458", ATTRS{idProduct}=="0001", ENV{ID_MM_DEVICE_IGNORE}="1", SYMLINK="bluegiga/bled112"
</pre>
<br />
And:<br />
<br />
<pre>$ cd ~/.wine/dosdevices
$ ln -s /dev/bluegiga/bled112 com1
</pre>
<br />
This way every time we plug out bled112 dongle into the computer, the correct device will be mapped under wine's com1.<br />
<br />
So with everything working like that, the only problem is how to write the resulting .hex file onto out ble112 device?<br />
<br />
The solution is cc-tool: <a href="http://sourceforge.net/projects/cctool/">http://sourceforge.net/projects/cctool/</a><br />
<br />
This tool isn't included in the distributions I know, so you have to download the sources and build it yourself. It's quite easy to do, it follows the usual configure-make-make install process. Just make sure you install the boost and libusb development libraries. If you use Ubuntu, you can do that by issuing the following command:<br />
<br />
<pre>$ apt-get install libboost-all-dev libusb-1.0-0-dev
</pre>
<br />
After this, you can download and install the cc-tool package. Open up a terminal window, and do the following:<br />
<br />
<pre>$ wget http://heanet.dl.sourceforge.net/project/cctool/cc-tool-0.26-src.tgz
$ tar -xf cc-tool-0.26-src.tgz
$ cd cc-tool
$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
checking for LIBUSB... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: executing libtool commands
$ make
CXX src/main.o
...
CXXLD cc-tool
$ sudo make install
make[1]: Entering directory `/home/netom/install/cc-tool'
...
make[1]: Leaving directory `/home/netom/install/cc-tool'
$ sudo cc-tool
CC Debugger device not found
</pre>
<br />
You have to run the cc-tool program as root to be able to access USB. The last line is an error message. It means that the cc-debugger is not connected.
When the device is connected, cc-tool recognizes it:
<br />
<pre>$ sudo cc-tool
Programmer: CC Debugger
Target: CC2540
No actions specified
</pre>
<br />
You can get some info on the target with the -t switch:
<br />
<pre>$ sudo cc-tool -t
Programmer: CC Debugger
Target: CC2540
Device info:
Name: CC Debugger
Debugger ID: 8561
Version: 0x05CC
Revision: 0x0034
Target info:
Name: CC2540
Revision: 0x20
Internal ID: 0x8D
ID: 0x2540
Flash size: 128 KB
Flash page size: 2
RAM size: 8 KB
Lock data size: 16 B
</pre>
<br />
For more info, just enter cc-tool --help.
<br />
Okay. Now everythin should work fine. Let's build the bkble112 thermometer demo.
Cd into the ble/example/bkble112 director, and issue the following commands. I'm going to use
absolute paths on my own machine. Replace them with your paths.
<br />
<pre>$ cd ble/example/dkble112
$ wine /home/netom/install/ble/bin/bgbuild.exe project.bgproj
baudm:216 baude:10 rate:57617
UART channel:0
baudrate :57600
actual :57617
error% :0.0295139
alternate f:2
ports:14336
C:/users/netom/Temp/qt_temp.crifX8
0
C:/users/netom/Temp/qt_temp.dflNl8
RAM Memory
-------------------------------------------------
Core RAM end @ 0x00cc5 3269
Top of RAM @ 0x01f00 7936
RAM left for data = 0x0123b 4667
Attribute RAM - 0x00007 7
Connections 1 - 0x00194 404
RAM for packet buffers 109 - 0x0109b 4251
Flash Memory
-------------------------------------------------
Core flash reserved @ 0x18000 98304
Top of flash @ 0x1f800 129024
Flash left for data = 0x07800 30720
Common configuration - 0x00070 112
16 bit UUIDs - 0x00022 34
128 bit UUIDs - 0x00000 0
Attribute database - 0x00084 132
Constant attributes data - 0x0007d 125
USB descriptor - 0x000c6 198
BGScript - 0x001f1 497
Flash for PS Store 14 - 0x07000 28672
$ sudo cc-tool -few out.hex -v
Programmer: CC Debugger
Target: CC2540
Erasing flash...
Completed
Writing flash (128 KB)...
Completed (1.25 s.)
Verifying flash...
Completed (0.54 s.)
</pre>
<br />
If you notice random errors, try programming without the -f (fast) switch.
<br />
You can build your own projects similarly to this one.
<br />
One more trick: if you are tired entering full paths and wine commands, create
and alias
<br />
<pre>$ echo 'alias bgbuild="wine <install dir>/ble/bin/bgbuild.exe"' >> ~/.bashrc
</pre>
<br />
After this, you only have to type bgbuild project.xml to build your project.
Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com13tag:blogger.com,1999:blog-4443103563245822857.post-71408816702847394342013-04-02T22:13:00.001-07:002014-06-27T12:36:04.049-07:00Speeding up bitcoin-qt on Linux<h2>
The problem</h2>
After freshly installing bitcoin-qt - the de-facto bitcoin client - to a computer, it starts downloading "blocks" from the network. The last few (tens of) thousands of blocks are especially slow, and the client is using the disk heavily.<br />
<br />
Examining the disk usage closely with iostat showed that the disk utilization was 100%, but at a very low data rate. My disk can read and write about 150 MBytes/second if utilized properly, but bitcoin-qt could only write about <b>one</b> megabytes per second.<br />
<br />
(EDIT: thanks for the donation, you rock! :) ) <br />
<br />
<a name='more'></a><br />
<h2>
The cause</h2>
Bitcoin-qt is very nicely written program that uses the guarantees granted by modern file systems to improve data integrity. This means that if you pull the plug in the middle of disk operations, your filesystem will still be able to come back to life after reboot, and will do so quite quickly.<br />
<br />
The program does this by calling the fsync() system call a lot. This ensures that when it returns, <i>every</i> byte sent to the file handle will be on the magnetic surface of your disk. Well, it seems to be the case in the default Ubuntu desktop configuration at least.<br />
<br />
The ext4 file system - what I use - sends so-called <i>barrier</i> commands to the disk when it fsync() is issued (well, on any journal commit). The barrier ensures that everything is on the disk before it starts to process commands after the barrier. By the time the barrier command returns, everything is on the spindles.<br />
<br />
<br />
<h2>
The solution</h2>
<br />
The block chain download is just painfully slow. Power outages are rare and I have backups. Data integrity therefore doesn't seem like a major concern to me, at least for a few hours while the block chain is downloading. If I could just disable barriers, that might improve performance.<br />
<br />
The examples below assume that you keep your block chain on the / partition. If you have a separate partition for /home, you probably have to use that. Anyway if you start tweaking your file system like this, I expect you to know how to figure this out. ;)<br />
<br />
This is easy to test, barrier writes can be disabled on-line:<br />
<br />
sudo mount -o remount,nobarrier /<br />
<br />
This disables barrier writes nicely <b>totally risking the data integrity of your filesystem, unless you're using battery-backed RAID</b>. The performance improvement of the client was drastic. The disk utilization dropped to about 50%, and my desktop programs (like firefox) was responsive again.<br />
<br />
Do this only while you're initially downloading blocks. This can be dangerous on the average desktop computer. <b>Always keep backups.</b><br />
<br />
When the client is finished downloading blocks, return to normal operation:<br />
<br />
sudo mount -o remount,barrier /<br />
<br />
<h2>
Donate</h2>
Oh, of course, you can always send me a few BTCs ;)<br />
<br />
Bitcoin: 19uhBv4n8R7aUcJSMsD6vkaLotfXwcqavY<br />
<br />
Litecoin: LdiZ2hwCsSh71CUK9zaac5iUS1KnQ5pS6DFábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-20320732375898784502013-03-29T02:24:00.002-07:002015-11-24T05:44:33.700-08:00Finding a Hamiltonian path - a randomized aproach<h2>
The problem</h2>
There is an <a href="http://ch24.org/">international programming</a> contest in Hungary held in every year. I like to participate, the problems are very entertaining.<br />
<br />
One of the problems in 2003 was an idealized DNA sequence assembly based on short reads. The sequences of course were generated by a computer (they weren't actually sequenced DNA data), and the input was very "clean": <br />
<ul>
<li>There were given N reads, each L long.</li>
<li>The sequences overlapping exactly at five bases (five characters)</li>
<li>There were no read errors, or any kind of noise</li>
</ul>
That's it. It felt tempting to build a graph of that data.<br />
<br />
Doing it in Python, the adjacency dictionaries (one forward, one reversed) was built in a couple dozen seconds for the largest files. (Python dictionaries are hash maps, they're pretty fast, and scale well.)<br />
<br />
<a name='more'></a><br />
<h2>
Possible solutions</h2>
Question is, how do I build a path in such a graph that contains all nodes exactly once?<br />
<br />
Such a path is called a Hamiltonian path, and finding one in an arbitrary graph is an NP-complete problem. This means that it is hopeless to come up with an efficient algorithm that finds such paths quickly for small graphs, and only reasonably slower in bigger graphs.<br />
<br />
Since the biggest input for this problem contained 10.000 nodes, it looked totally hopeless to exhaustively search through all possible paths, it would take a very-very long time (sun-would-turn-into-a-red-giant long).<br />
<h2>
Backtrack </h2>
...is the name of the process that would be still running under a red sun (worst case). It would also take insanely lot of memory to work.<br />
<br />
However, if the graphs would have only a very few (like 1, or two) connections from a node towards other nodes (few branches along the possible paths), then it would work OK for even a few thousand nodes.<br />
<h2>
Building every possible Hamiltonian path</h2>
The idea is to start from a set of paths that contain only one node. Create a set and put a one-long path for every node.<br />
<br />
In each iteration:<br />
<ul>
<li>Try to extend the paths in every way, and build a new set from the extensions.</li>
<li>If it cannot be extended, it won't be in the new set.</li>
<li>When the new set is ready, discard the old one.</li>
</ul>
Do this until you will have a path that is Hamiltonian.<br />
<br />
The problem with this is that the number of paths will be very large after 6-8 steps, and will eat many gigabytes of memory. Handling such a data structure is also very slow.<br />
<h2>
Discarding paths</h2>
Since, we only need a single path, we can try to get rid some of the paths.<br />
<br />
I found out that if I randomly throw away paths so that only a few thousand remain after every iteration, I can solve the problems under one thousand nodes.<br />
<br />
This is because there are many solutions, and we have a reasonable chance to find one even if we throw some candidates away.<br />
<h2>
Random path extension</h2>
The algorithm above is really just a complicated way of building random paths. Random path building can be done more quickly - therefore it's faster. We start from a random node, and add a random neighbour. If we can't add more, we check if the path is Hamiltonian. If it isn't, we start again.<br />
<br />
This algorithm below can solve the problem for every input in reasonable time.<br />
<br />
The problem is, that sometimes it takes two minutes to solve the hardest input, and sometimes it takes half an hour. Good thing is that it's very easy to parallelize: just run the same program on every processor core you have. One of them will find the solution quicker than the others.<br />
<br />
<br />
It is surprising how a simple algorithm like this can solve such a hard problem.<br />
<br />
Never underestimate the power of simple randomized algorithms!Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-91757166595569421042013-01-17T02:18:00.002-08:002013-10-26T07:45:37.183-07:00ERROR: "Problem with InfiniDB process PrimProc, should be a single version running"<h2>
InfiniDB</h2>
Infinibd is a column-oriented database (actually a MySQL storage engine, or to be more precise, a MySQL distribution with this special storage engine). It can handle analytical / reporting queries very well.<br />
<br />
<h2>
ERROR</h2>
I installed it on my laptop to form a nice data warehouse / reporting development environment together with <a href="http://community.pentaho.com/">Pentaho's</a> tools. It did OK for a while, but it seized to work about a week ago.<br />
<br />
Upon startup, the /etc/init.d/infinidb script gave this error message:<br />
<br />
Problem with InfiniDB process PrimProc, should be a single version running<br />
<br />
<a name='more'></a><br /><br />
<h2>
DEBUG </h2>
Since I saw no PrimProc in the process list, I thought the error message might be a little off. I found the place where it counted various processes in the process list, including PrimProc. I echo-ed the result, and I saw the there were 0 instances running. Another example for a confusing error message.<br />
<br />
I took a look into the file /vat/log/Calpont/crit.log and I saw this:<br />
<br />
Jan 7 16:01:14 sphynx PrimProc[16613]: 14.857119 |0|0|0| C 28 CAL0045: FATAL ERROR: PrimProc has allocated too much memory! PrimProc is restarting.<br />
<br />
PrimProc is InfiniDB's Primitive Processor. It executes the smallest units of called primitives.<br />
<br />
<h2>
FIX</h2>
The amount of memory dedicated to the data buffer cache utilized by PrimProc is defined in /usr/local/Calpont/etc/Calpont.xml. It is called <b>NumBlocksPct</b>. The value of the variable is in percent of the phisical memory, and defaults to a very high value, and it can prevent PrimProc from starting, or it can cause heavy swapping. <b>Since I do not use swap, PrimProc can't start</b> with xorg, firefox, openoffice, thunderbird, skype, ... running.<br />
<br />
The solution was to set NumBlocksPct to a lower value, in my case 20 is plenty.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-19372399797033987982013-01-17T01:55:00.001-08:002013-10-26T07:46:59.655-07:00Implementing a Hilbert (90 degree shift) filter in Python<h2>
Why?</h2>
A digital 90° phase shift filter is an important building block of the so-called Software Defined Radios (SDRs).<br />
<br />
And SDR is a radio that has (relatively) minimal hardware, and most of the features are implemented in software. There are no fancy buttons and displays, but there is a UI application that controls the box. The box is a direct conversion receiver. It just converts part of the radio frequency spectrum to the audio spectrum (the reality is a bit more complicated, but you got the idea).<br />
<br />
An SDR usually provides two signals that are almost the same, except one signal's frequency components are shifted 90 degrees in one direction.<br />
<br />
Given these two audio signals, the software can do <i>anything</i> a conventional receiver can achieve with bulky components or expensive integrated circuits. Plus, the software can be changed easily. Actually this is the greatest thing in SDRs.<br />
<br />
The software can demodulate AM signals by LW-MW-SW broadcast stations, or FM usually encountered in the URH spectrum. It can make amateur SSB and CW signals readable, or can decode digital transmissions like DAB. Anything you can imagine.<br />
<br />
<a name='more'></a><br /><br />
<h2>
SSB reception</h2>
I'm mainly interested in amateur (HAM) radio, and so I'd like to receive SSB transmissions. To do that, I have to 90° phase-shift one of the incoming audio signals from the SDR box, and sum with the other. The result is an SSB receiver.<br />
<br />
<h2>
FIR filters</h2>
I use the numpy.filter Python package to process audio signals. The firwin and firwin2 function are very useful for designing all sorts of FIR filters, but I could not find a built-in function that can readily be used to shift all frequencies by 90 degrees.<br />
<br />
A filter is represented by its <i>coefficients</i>. The simplest FIR filter one can imagine is the averaging filter. You take N samples, add them, and divide the result by N. If you encounter a new sample, you just drop one sample from one end of the line, and put the new sample to the other end. You add them and divide by N again. Simple. The general FIR filter is a weighted averaging filter. The coefficients are the weights.<br />
<br />
The coefficients are also the filter's <i>impulse response</i>. That is, if you filter a signal that has all zero samples except one, you get all zeroes and the coefficients somewhere in the middle. Example:<br />
<br />
Signal:<br />
0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 ...<br />
<br />
Coefficients:<br />
0.3 0.4 0.3<br />
<br />
Response:<br />
<br />
0 0 0 0 0 0 0 0 0 0.3 0.4 0.3 0 ...<br />
<br />
As you can see, the response contains the coefficients.<br />
<br />
This is important because the impulse response can be calculated from the desired frequency response by the <i>inverse discrete Fourier Transform</i>.<br />
<br />
<h2>
90° phase shift</h2>
In our case, this leads us to the code below:<br />
<br />
<pre class="brush: python">import numpy.fft
import matplotlib.pyplot as plt
coeffs = numpy.fft.fftshift(
numpy.fft.ifft([0]+[1]*100+[0]*100)
)
plt.plot(numpy.imag(coeffs))
plt.plot(numpy.real(coeffs))
plt.show()
</pre>
<br />
The variable <i>coeffs</i> holds the filter coefficients. It can be used with the numpy.signal.lfilter function to process data.<br />
<br />
The code works by calculating the inverse discrete Fourier Transform of a strange frequency response. It reads like this: "pass every negative frequencies, and supress all of the positive frequencies". This series has a complex iDFT. We are interested in the imaginary part of this inverse. The fftshift function rearranges the result. The plt.* function calls shows us the result.<br />
<br />
Note that the filter coefficients are complex. An audio signal from and SDR receiver can be processed by taking the left and rigth channel as the real and imaginary parts of a complex signal, and filtering that - also complex - signal. The filtered signal will also be complex, and the imaginary part will be shifted by 90° (at every frequency).<br />
<br />
If the signal to be filtered is real-valued, then the resulting filtered signal will still be a complex one: a real, and a 90°-shifted imaginary.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-66778963102407494572013-01-09T03:12:00.001-08:002013-10-26T07:47:27.359-07:00Converting a dual-boot MBR partitioned disk to GPT<h2>
Background story</h2>
I always liked computer games. My first PC was a Pentium 100 MHz machine with very little RAM, which was dedicated almost entirely to a RAM disk. My first precious PC was lacking a hard drive. Every time I wanted to use it, I had to boot up from a floppy and copy software from a few other floppies to the RAM disk. One of my favourite games was UFO: Enemy Unknown. It took a lot of time to boot and copy everything in place, so my machine was on most of the time, and I played a lot on it :) This kinda shows my commitment to PC games.<br />
<br />
The famous game was recently rebooted by Firaxis as "XCOM: Enemy Unkown". The game was kinda sluggish on my current setup, so I decided to buy new hardware.<br />
<h3>
<a name='more'></a> </h3>
<h2>
The new setup</h2>
<ul>
<li>Asrock X79 Extreme6 GB motherboard (with UEFI)</li>
<li>INTEL Core i7-3820 3.60GHz LGA-2011 processor</li>
<li>2 Corsair Vengeance 16GB quad kits, 2400 MHz</li>
<li>2 SEAGATE Barracuda ST3000DM001 (3000GB, 7200rpm 64MB cache, SATA3)</li>
<li>New PSU, new chassis, display, graphics card, you name it. </li>
</ul>
After <b>long </b>debugging, involving a <a href="https://bugs.launchpad.net/ubuntu/+source/memtest86+/+bug/1071209">Memtest86+ bug</a> it turned out that two identical RAM kits might not work together on full speed, and one kit is flawed, anyway, RMA in progress.<br />
<h3>
</h3>
<h2>
Always keep two copies. And a few backups too.</h2>
I always take backup seriously. Programming and fiddling with hardware and software quickly became a profession of mine. I cannot afford to lose my data.<br />
<br />
The old setup had one 2TB fixed HDD and a 640GB HDD that was lying in the drawer, holding backups. Important git repositories was also on the <a href="http://www.vengit.com/en/">Company's</a> dev server, and in S3 too.<br />
<br />
The drawback of this solution is downtime. If the fixed disk (holding the OS-es) would have failed, I would have to reinstall everything.<br />
<br />
This is why I decided to buy two identical hard disks. My intention was to keep two copies of the whole system(s), so if one fails, I can always plug in the other. No precious time wasted. A new disk and on overnight dd can restore redundancy.<br />
<h3>
</h3>
<h2>
Moving the data to the new disks</h2>
Actually "overnight dd" part was when I discovered that something is terribly wrong with the new rig, and the three day nightmare of memtest86+ debugging and data copying started.<br />
<br />
Finally, I managed to copy the 2TB old disk to both 3TB disks. Both could be booted up.<br />
<br />
I have two operating systems: an Ubuntu Precise for fun & profit, and a Windows 8 for fun only (yes, games). Both systems could be booted up at this point. I use Grub 2.<br />
<br />
Too bad the old disks used old MBR partition tables. This kind of partition table can't handle disks larger than about 2 Terabytes, so at this point I had absolutely no use of the 1 extra terabyte at the end of the disk. What a waste of money. <br />
<h3>
</h3>
<h3>
</h3>
<h2>
Converting a disk to GPT</h2>
To do this, I booted from one HDD, and left the other one in an eSATA rack.<br />
<br />
The conversion can be done with the gdisk (for GPT fdisk) software from ubuntu. It can be installed like this:<br />
<br />
apt-get install gdisk<br />
<br />
To convert an MS-DOS partition table to GPT, just launch gdisk, and hit "w" to write the new, converted GPT to the disk. The data will remain untouched, however, the disk will become unbootable.<br />
<br />
I sorted the partitions so that their entries in the partition table were the same as they appear on the disk, so the partition numbering was messed up beyond recognition. The data is still there, but the OSes could not boot up due to several equally forbidding reasons.<br />
<br />
I used gparted tool to resize and move the partitions. Windows got almos 1TB, and Linux got the rest. <br />
<br />
Converting a disk from MBR to GPT _and_ having Windows to boot up without a reinstall is pretty much impossible with my little knowledge on Windows. The terrible documentation and limited range of tools makes the situation even worse, so I decided to completely reinstall Windows 8 Pro. I've lost no data, since the other disk, and the old disks held the old Windows partition, and I only used it to play with stuff downloaded from Steam, so everything "important" was also in the cloud.<br />
<br />
<br />
<h2>
Installing Windows on GPT with UEFI</h2>
<br />
Windows can only be installed on GPT if it is booted up in UEFI mode.<br />
<br />
I realized that the Windows 8 Pro update disk I've bought is only willing to boot up in legacy (a.k.a. BIOS mode). Darn.<br />
<br />
The problem is that the CD is constructed in such a way, that it is <i>almost</i> possible to boot up in UEFI mode. The files are there, but the CD's filesystem does not allow UEFI to boot from it. If it would be FAT, it could boot. <b>The solution is to copy the entrire Windows 8 Pro install cd onto a FAT formatted flasdrive.</b> After doing so, Windows started happily in UEFI mode, and even run faster from the quick little thumbdrive.<br />
<br />
Windows need a few extra partitions on GPT to be happy. Actually UEFI itself requires a FAT32 partition to be able to boot (this is why the thumbdrive was needed in the first place).<br />
<br />
In the Windows installer, I removed the Windows partition, and told Windows to install itself on the resulting "free space". It figured out the situation, and created the UEFI system partition (the stuff for booting wit the FAT filesystem), a partition for recovery, one MSR for it's private business, and a data partition for the actual filesystem mortals use for their everyday business.<br />
<br />
Installation went smoothly, Windows finally booted up form the disk, in UEFI. Now was time to fix Ubuntu.<br />
<br />
<h2>
Installing GRUB 2 in UEFI mode</h2>
In Ubuntu, I had to create a /boot/efi mount point, and mount the UEFI system partition, uninstall the old grub2, and install grub-efi.<br />
<br />
To install grub-efi, one has to be boot the system in UEFI mode. What?!<br />
<br />
This chicken-and-egg problem can be solved with a recent Ubuntu desktop live CD by setting the UEFI setup to boot the DVD in UEFI mode. Yepp, Ubuntu's install CD can boot up in UEFI mode no problem, Windows can't. +1 for Ubuntu again.<br />
<br />
To install grub, one has to mount the / partition under somewhere, like /mnt. After chroot-ing into this partition, /proc, /dev, /dev/pts, /sys and the aformentioned /boot partition have to be mounted with mount <mount point> commands. After this, a grub-install writes the necessary files onto the UEFI boot partition, and update-grub writes grub.cfg.<br />
<br />
<h2>
Chainloading the Windows UEFI boot loader from Grub-EFI</h2>
To be able to do this, I followed the advice at <a href="http://askubuntu.com/questions/193144/dual-boot-uefi-windows-7-and-ubuntu-12-04-both-64-bits-w7-entry-doesnt-appea">http://askubuntu.com/questions/193144/dual-boot-uefi-windows-7-and-ubuntu-12-04-both-64-bits-w7-entry-doesnt-appea</a>. Basicly:<br />
<br />
<pre><code>grub-probe --target=fs_uuid /boot/efi/efi/Microsoft/Boot/bootmgfw.efi</code></pre>
<br />
This will output an UUID. Take a note of this.
<br />
Add this to <code>/etc/grub.d/40_custom</code>:<br />
<br />
<pre><code>menuentry "Windows 8" {
search --fs-uuid --no-floppy --set=root UUID
chainloader (${root})/efi/Microsoft/Boot/bootmgfw.efi
}
</code></pre>
<br />
And run update-grub after this.
<br />
<h2>
Fixing the boot order</h2>
<br />
Windows and Ubuntu create entries in the motherboard's NVRAM for their boot managers. You can select the default operation system at the UEFI setup (by quickly pressing something like DEL, or F1, or F2 when the system starts).<br />
<br />
However, whenever Windows starts, it makes itself the default operating system in that list. So after every Windows session, Windows will start again by default on the next boot. Nice huh?<br />
<br />
No, it isn't. Let's solve this.<br />
<br />
According to Microsoft's support forum, <i>it's impossible to make Windows to accept itself other than the first place on the boot list</i>.<br />
<br />
The solution is to create a batch file that runs at every bootup, and makes GRUB2 the default entry in the NVRAM.<br />
<br />
This can be done by opening up the group policy editor (search this in Metro UI), and navigation to the startup scripts section.<br />
<br />
The .bat file should look like this:<br />
<br />
bcdedit /set {fwbootmgr} DEFAULT {appropriate UUID}<br />
<br />
Where "appropriate UUID" is the UUID in GRUB, and can be read from the list that "bcedit /enum firmware".<br />
<br />
So Windows will make itself default, just to undo int a bit later when this file runs. Not a pretty solution, but it works.<br />
<br />Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-87024799379855638762013-01-09T02:27:00.001-08:002013-10-26T07:47:57.550-07:00ERROR: Pentaho Data Integration (Kettle) process runs twice<h2>
The Problem</h2>
I work a lot with <a href="http://www.pentaho.com/explore/pentaho-data-integration/">Pentaho Data Integration</a> a.k.a <a href="http://kettle.pentaho.com/">Kettle</a> toolkit. For those who don't known: Kettle allows you to build processes with a GUI that can be run in the IDE or from the command line, and reads data, converts and transforms it, then spits it out. It can deal with a lot of databases and various file formats, can invoke shell scripts, can run JavaScript snippets, and perform various conversions and transformations. Very handy when you have to load large, broken CSV files into relational databases just to mention an example.<br />
<br />
I recently re-designed one of our processes (or "jobs" in Kettle) when something really strange showed up during testing.<br />
<br />
Part of the process run twice. It seemed like I'd duplicated the whole process from some point.<br />
<br />
The IDE is basicly allows you to build a graph, the nodes are the process steps, the edges are telling what to do when a node was finished.<br />
<br />
Clearly, beside error handling edges, I only draw a single edge from any node. The thing I saw could only happened If I would drew two edges between some of the two nodes. The IDE don't even allow such thing to happen. Or do it?<br />
<br />
I tried to disable the edges (called "hops" in Kettle job terminology), to catch where the process goes two ways.<br />
<br />
I found out that disabling one of the hops did NOT cause the job to stop at that point - which is the thing it should have done. Instead, the script run correctly, executing it's tail only once.<br />
<br />
<a name='more'></a><br />
<h2>
The Solution</h2>
Something was clearly off the rails. In the IDE, I disabled one hop, but the command line tool still thought that there is an enabled hop between the two steps. When I enabled the hop, the command line tool probably saw TWO hops between the same nodes - a thing that is impossible to (intentionally) achieve in the IDE (which is called Spoon btw). I was suspecting file corruption, and/or serious command line tool bug.<br />
<br />
Kettle keeps the job description in XML files, therefore they can be opened and easily modified in a simple text editor. So this is exactly what I did: I fired up Geany, and opened the offending job file. This is what I saw:<br />
<br />
<br />
<pre>...
<hop>
<from>Set up dimension tables in stage</from>
<to>transform_fixlogs_to_stage</to>
<from_nr>0</from_nr>
<to_nr>0</to_nr>
<enabled>Y</enabled>
<evaluation>Y</evaluation>
<unconditional>N</unconditional>
</hop>
<hop>
<from>Set up dimension tables in stage</from>
<to>transform_fixlogs_to_stage</to>
<from_nr>0</from_nr>
<to_nr>0</to_nr>
<enabled>Y</enabled>
<evaluation>Y</evaluation>
<unconditional>N</unconditional>
</hop>
...
</pre>
<br />
Yep, there are TWO hops between the same nodes.<br />
<br />
Removing one of the hops with the text editor solved the problem, the job now runs correctly.<br />
<br />Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-24368255912992302422012-07-27T02:24:00.001-07:002013-10-26T07:48:30.352-07:00Bash script that cannot be run more than once at the same timePeople dealing with cron, and bash scripts that might take a bit longer than they're supposed to often encounter the following behaviour.<br />
<br />
Suppose that you launch something from cron in say every hour. The stuff usually completes in 10 minutes, but sometimes, when the load peaks, or network clogs, the process is running much slower. After an hour an other one is launched, further hogging the resources of the machine and possibly messing up data.<br />
<br />
The solution for this is to pay attention not to run twice of course (and also to fix the underlying problem that causing the slowdowns).<br />
<br />
<a name='more'></a><br /><br />
Craig Andrews posted an almos working shell script in his blog (<a href="http://candrews.integralblue.com/2009/02/one-instance-at-a-time-with-pid-file-in-bash/">http://candrews.integralblue.com/2009/02/one-instance-at-a-time-with-pid-file-in-bash/</a>)<br />
<br />
I hereby shamelessly re-post the snippet. My only excuse is that I've fixed the typo causing the output of kill to be written into a file named 1. Well, I also added a few comments and hints. Anyway. Here it is:<br />
<br />
<br />
<pre>#!/bin/bash
# Replace this with a meaningful, and unique filename
# You probably need root privileges to write /var/run.
# You can use any filename and path here for testing or
# for whatever reason.
pidfile=/var/run/your_solution_name.pid
if [ -e $pidfile ]; then
pid=`cat $pidfile`
if kill -0 2>&1 > /dev/null $pid; then
echo "Already running"
exit 1
else
rm -f $pidfile
fi
fi
echo $$ > $pidfile
# Do your stuff here.
# For testing purposes, we're just gonna
# sleep for 10 seconds. Try opening two
# terminal windows and launch the script
# in both one at the same time.
sleep 10
rm $pidfile
</pre>
Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-68998276212948359402012-07-17T05:55:00.001-07:002013-10-26T07:48:51.798-07:00How NOT to write SLOW programs with python and numpy<strike><b>How to write fast programs with numpy and python</b></strike> <br />
<br />
At first, I wanted to write a post about "how to write fast programs with python and numpy". After writing a few test cases it quickly turned out that the fast, numpy versions of tests are not only fast because of numpy, but because they use numpy "the right way". Let's see what does it mean.<br />
<br />
<a name='more'></a><br /><br />
<b>The problem</b><br />
<br />
I was writing a program that read a sound file, subtracted the right channel from the left (or the other way around, it doesn't matter), and played the result back as a single mono audio stream.<br />
<b><br /></b>
The point was to generate karaoke track out of a "normal" one by stripping the vocals. Since the vocals are often present equally and in-phase on both channels, after the subtraction no vocals was heard. Most of the time. This method has a drawback of ruining much of the other sounds in a track, but at that point I could live with that.<br />
<br />
<b>The slow solution</b><br />
<br />
<pre class="brush: python"># -*- coding: utf-8 -*-
import wave
import pyaudio
import numpy
f = wave.open('e.wav')
samples = numpy.fromstring(f.readframes(
f.getnframes()), dtype=numpy.int16)
print "Removing voice..."
S = numpy.empty(len(samples)/2, dtype=numpy.int16)
for i in xrange(len(samples)/2):
S[i] = samples[2*i]/2 - samples[2*i+1]/2
f.close()
print "Playing..."
p = pyaudio.PyAudio()
stream = p.open(
output_device_index = 0, format = pyaudio.paInt16,
channels = 1, rate = 44100, input = False,
output = True, frames_per_buffer = 4410
)
for i in xrange(len(samples)/44100):
stream.write(S[i*4410:(i+1)*4410].tostring())
stream.close()
</pre>
<br />
<b>The fast solution</b><br />
<br />
The infinitely faster solution only differed in a few lines after the line that prints "Removing voice...":<b> </b><br />
<br />
<br />
<pre class="brush: python">print "Removing voice..."
S = samples[0::2]/2 - samples[1::2]/2
print "Playing..."
</pre>
<br />
<b>Digging deeper </b><br />
<br />
Using the built-in compile() function, and the wonderful dis module, let's examine what's happening in both cases.<br />
<br />
Slow case: <br />
<br />
<pre>>>> c = compile("for i in xrange(len(samples)): S[i] = samples[2*i]/2 - samples[2*i+1]/2", "dummy", "exec")
>>> dis.dis(c)
1 0 SETUP_LOOP 68 (to 71)
3 LOAD_NAME 0 (xrange)
6 LOAD_NAME 1 (len)
9 LOAD_NAME 2 (samples)
12 CALL_FUNCTION 1
15 CALL_FUNCTION 1
18 GET_ITER
>> 19 FOR_ITER 48 (to 70)
22 STORE_NAME 3 (i)
25 LOAD_NAME 2 (samples)
28 LOAD_CONST 0 (2)
31 LOAD_NAME 3 (i)
34 BINARY_MULTIPLY
35 BINARY_SUBSCR
36 LOAD_CONST 0 (2)
39 BINARY_DIVIDE
40 LOAD_NAME 2 (samples)
43 LOAD_CONST 0 (2)
46 LOAD_NAME 3 (i)
49 BINARY_MULTIPLY
50 LOAD_CONST 1 (1)
53 BINARY_ADD
54 BINARY_SUBSCR
55 LOAD_CONST 0 (2)
58 BINARY_DIVIDE
59 BINARY_SUBTRACT
60 LOAD_NAME 4 (S)
63 LOAD_NAME 3 (i)
66 STORE_SUBSCR
67 JUMP_ABSOLUTE 19
>> 70 POP_BLOCK
>> 71 LOAD_CONST 2 (None)
74 RETURN_VALUE
</pre>
<br />
Fast case:<br />
<br />
<pre>>>> import dis
>>> c = compile("S = samples[2::0]/2 - samples[2::1]/2", "dummy", "exec")
>>> dis.dis(c)
1 0 LOAD_NAME 0 (samples)
3 LOAD_CONST 0 (2)
6 LOAD_CONST 1 (None)
9 LOAD_CONST 2 (0)
12 BUILD_SLICE 3
15 BINARY_SUBSCR
16 LOAD_CONST 0 (2)
19 BINARY_DIVIDE
20 LOAD_NAME 0 (samples)
23 LOAD_CONST 0 (2)
26 LOAD_CONST 1 (None)
29 LOAD_CONST 3 (1)
32 BUILD_SLICE 3
35 BINARY_SUBSCR
36 LOAD_CONST 0 (2)
39 BINARY_DIVIDE
40 BINARY_SUBTRACT
41 STORE_NAME 1 (S)
44 LOAD_CONST 1 (None)
47 RETURN_VALUE </pre>
<br />
There are differences that are quite obvious at first sight. The slow version has a loop that executes a bunch of operations like BINARY_ADD, BINARY_DIVIDE, etc. in each iteration, so there's O(n) python-related stuff to execute.<br />
<br />
In the fast case, there are a fixed amount code to run, so we only have O(1) python overhead. The lion's share is written in C. Although it's easy to write slow code in C, numpy is certainly not slow.<br />
<br />
The conclusion is that if you're programming with numpy, you're better avoid iterating over numpy arrays and doing stuff on them. Numpy has a great many built-in functions that cover everything came into my mind since I've started working with it. Reinventing the wheel is highly counterproductive.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com0tag:blogger.com,1999:blog-4443103563245822857.post-9466672726319417422012-01-24T14:01:00.000-08:002013-10-26T07:49:06.633-07:00Thrift / c_glib and Cassandra<b>Thrift </b><br />
<br />
<a href="http://thrift.apache.org/">Thrift</a> is apache's tool. It can generate client / server codes based on a file written in it's own descriptor language.<br />
<br />
At first I was thrilled how easy it'll be to write a Cassandra client with it: "you just have to generate the C files, #include them, call a few functions and it's done".<br />
<br />
Yeah. Like anything in the world works like that. And this particular thing is no exception.<br />
<br />
<a name='more'></a><br /><br />
Thrift comes with a documentation that is... wait! It doesn't really comes with any documentation at all. The stuff that's in the package and / or scattered on the Net in the form of blog posts and bug reports is outdated and only can be used to prevent the enemy from using this great weapon.<br />
<br />
My last expression isn't a sarcastic one, thrift would be great if I could wield it correctly.<br />
<br />
<b>Cassandra</b><br />
<br />
<a href="http://cassandra.apache.org/">Cassandra </a>is <a href="http://en.wikipedia.org/wiki/NoSQL">noSQL</a> database. It doesn't really matter now how it works exactly, it's enough if you know that one can store and fetch data with it, and can connect to it over the network.<br />
<br />
Coincidentally, it uses Thrift to describe it's interface, so people of different sex, religion and programming language can generate their own interface libraries. First, I tried to put together a client in C++ based on <a href="http://posulliv.github.com/2010/02/22/cpp-cassandra.html">this</a> article. Cassandra, Thirft and gcc evolved somewhat since 2010, and / or I might be using an exotic combination of software (Ubuntu Oneiric), or the Gods might be angry at me for some strange reason, or I may be simply too dumb to follow a bit outdated tutorial solving a few problems along the way; anyway I could not get the code compiled.<br />
<br />
<b>C and GLib</b><br />
<br />
I have much more experience with C than C++, so I decided to throw the C++ code away my co-worker has been writing, and start from scratch with C. I was prepared to read and interpret the Thrift interface descriptor file with my already melting brain, and write the C code myself.<br />
<br />
As I started to work I discovered that Thrift CAN generate C interface libraries. It is a bit incomplete in 0.8.0, since it does not generate the server skeleton file; it didn't really matter for me.<br />
<br />
I cd'd into Cassandra's <span style="font-family: "Courier New",Courier,monospace;">interface</span> directory and issued <span style="font-family: "Courier New",Courier,monospace;">thrift -gen c_glib cassandra.thrift<span style="font-family: inherit;"> command, just to find the generated sources under </span>gen-c_glib </span>directory.<br />
<br />
The sources was clean and readable despite the fact that they were auto-generated.<br />
<br />
I even found a <a href="https://issues.apache.org/jira/browse/THRIFT-976">small example</a>, and it compiled OK.<br />
<br />
I had to replace a few lines, to work with Cassandra instead of the calculator example. The following is the re-write for Cassandra, with connecting to a server on localhost on the default port, and executing a query that fetches a value from keyspace "example", column family "examplecf", with key "foo", from the "bar" column. Error handling might be incomplete.<br />
<br />
Warning: I don't know a thing a about glib, and I suspect that the code below is NOT the way to use it. It works here though. I maybe will improve it in the distant future. <br />
<br />
<pre class="brush: c">#include <stdio .h="">
#include <string .h="">
#include "gen-c_glib/cassandra.h"
#include "protocol/thrift_protocol.h"
#include "protocol/thrift_binary_protocol.h"
#include "transport/thrift_framed_transport.h"
#include "transport/thrift_transport.h"
#include "transport/thrift_socket.h"
#include "gen-c_glib/cassandra.h"
int main(int argc, char** argv) {
ThriftSocket *tsocket;
ThriftTransport *transport;
ThriftProtocol *protocol;
CassandraClient *client;
CassandraIf *service;
InvalidRequestException *ire = NULL;
NotFoundException *nfe = NULL;
UnavailableException *ue = NULL;
TimedOutException *te = NULL;
ColumnOrSuperColumn *result;
GError *error = NULL;
GByteArray column = {
.data = (unsigned char *)"bar",
.len = 3
};
ColumnPath *cp = NULL;
GByteArray key = {
.data = (unsigned char *)"foo",
.len = 3
};
g_type_init();
tsocket = THRIFT_SOCKET(
g_object_new(
THRIFT_TYPE_SOCKET, "hostname",
"localhost", "port", 9160, 0
)
);
transport = THRIFT_TRANSPORT(
g_object_new(
THRIFT_TYPE_FRAMED_TRANSPORT, "transport", tsocket, 0
)
);
protocol = THRIFT_PROTOCOL(
g_object_new(
THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, 0
)
);
client = CASSANDRA_CLIENT(
g_object_new(
TYPE_CASSANDRA_CLIENT, "input_protocol",
protocol, "output_protocol", protocol, 0
)
);
service = CASSANDRA_IF(client);
if (
!thrift_transport_open(transport, 0) ||
!thrift_transport_is_open(transport)
) {
printf("Could not connect to server\n");
return 1;
}
printf("Connected to cassandra at localhost:9160\n");
cassandra_client_set_keyspace(
service, "example", &ire, &error
);
if (ire) {
printf("Invalid request exception: %s\n", ire->why);
return 1;
}
if (error) {
printf("An error has occured\n");
return 1;
}
printf("Selected keyspace example\n");
cp = g_object_new(TYPE_COLUMN_PATH, 0);
cp->column_family = "examplecf";
cp->column = &column;
cp->__isset_column = TRUE;
cassandra_client_get(
service, &result, &key, cp, CONSISTENCY_LEVEL_QUORUM,
&ire, &nfe, &ue, &te, &error
);
if (ire) {
printf("Invalid request exception: %s\n", ire->why);
return 1;
}
if (nfe) {
printf("Row not found\n");
return 1;
}
if (ue) {
printf("Unavailable exception\n");
return 1;
}
if (te) {
printf("Timed out exception\n");
return 1;
}
if (error) {
printf("An error has occured\n");
return 1;
}
printf(
"The result is %s\n",
strndup(
(char *)result->column->value->data,
result->column->value->len
)
);
/* Don't forget to free resources if
* your program runs longer than this */
return 0;
}
</string></stdio></pre>
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Arial,Helvetica,sans-serif;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Arial,Helvetica,sans-serif;">I compiled thecode with the following commands:</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span></span>
<div style="font-family: "Courier New",Courier,monospace;">
gcc -c `pkg-config --cflags thrift_c_glib` test.c -o test.o</div>
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
gcc -c `pkg-config --cflags thrift_c_glib`\</div>
<div style="font-family: "Courier New",Courier,monospace;">
gen-c_glib/cassandra.c -o cassandra.o</div>
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
gcc -c `pkg-config --cflags thrift_c_glib`\</div>
<div style="font-family: "Courier New",Courier,monospace;">
gen-c_glib/cassandra_types.c -o cassandra_types.o </div>
<div style="font-family: "Courier New",Courier,monospace;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
libtool --tag=CC --mode=link gcc `pkg-config --libs thrift_c_glib` -o test test.o cassandra.o cassandra_types.o</div>
<br />
The last command is even more cryptic then the others, so here's the explanation: <br />
<br />
The pkg-config command is used to query for compilation flags of program that use installed libraries. It's the library's <span style="font-family: "Courier New",Courier,monospace;">make install</span> script's responsibility to install this info. If a package is installed from the repository of your distribution, this information is installed by the package manager. The rest of the command line should be clear.<br />
<br />
UPDATE:<br />
<br />
Note that the key, column name and value does NOT contain the trailing zero byte.Fábián Tamás Lászlóhttp://www.blogger.com/profile/02408802258485717905noreply@blogger.com7