How to embed a Xively chart image

Note: I don’t do this any more. Third party APIs are a world of hurt – you save some time up front and end up chasing someone else’s upgrade path on their agenda, supporting their ads. There’s nothing fundamentally wrong with building cloud services and dependencies into your work, as long as you don’t want it to be up there for more than six weeks in my experience.

In their increasing commercialisation as Pachube went away from the original hacker ethos to Cosm, and now Xively, Xively have totally borked the documentation, at least for freebie cheapskates.

The most obvious thing you want to do with an IoT device is to chart the time series. With Pachube and Cosm that was easy. It’s also pretty easy using Xively, but what isn’t easy is finding out how to do it!


I don't use Xively any more so this is static
I don’t use Xively any more so this is a static capture from some time ago

This is what I’m after. I have a Geiger counter, and a PIC ticks up a counter each time it detects an event. After a minute is up, it reads the counter, sends it via LLAP over a serial RF network to a Raspberry Pi that uploads the count to Cosm Xively and resets the counter, and it all starts over again.

Note: I’ve now moved to running RRDtool on the Pi because I found Xively just a little bit too intermittent, plus there’s the whole getting control of my own data thing. I found it more reliable to store the data locally, generate the graph and then upload the graph. It isn’t necessarily Xively’s fault – I get the feeling  my ADSL connection just isn’t reliable enough for something that needs to update every minute,

RRDtool looks a bit more grungy, though I kinda like that here –

A little bit more grungy from RRDtool, but more reliable in my application
A little bit more grungy from RRDtool, but more reliable in my application

Anyway, back to Xively, and how easy graphing from Xively worked for me –

It was easy to find out how to do it on Pachube and Cosm but because Xively is all about provisioning IoT bits rather than hackers’ web charting now it’s the devil’s own job to find out how to do it. Xively favour stackexchange for support nowadays, but they can be less than helpful to n00bs or those who do not show that they feel The Force to the required extent 😉

Here is what I did to embed a Xively chart:

Read this from Xively on how to embed a chart – it’s not linked from any obvious place.

It so happens that the png chart option is a specific instance of reading a single datastream. You, me, and everybody else asks Google ‘how do I embed a Xively chart image’. Xively think of this in terms of ‘how do I display as single datastream’ and they’d rather you not do that, but parse the data returned from the feed, so they hide the relevant page in a backwater of their documentation because they don’t see it as important. Which is entirely their right – after all, you get the support you pay for!

Then import the image like so

obviously fiddle with width height, colour to taste. Note that it doesn’t need you API KEY, because presumably a PNG isn’t about to start posting data back to your feed any time soon. One of the more useful extra items is the duration=3hours

which doesn’t happen to be one of the official parameters in the Xively historical parameters documentation, but it is in the example they give and is a useful undocumented feature. A shorter duration works better with the Geiger count.

Your device needs to be public access if you want other people to see it, natch 😉 Make it private and they will be invited to log in on Xively, which sort of defeats the point.

Now why did that have to be so hard?

Alternatives ways of showing Xively data

The embedded Xively chart is loaded when you load your page, it doesn’t auto-refresh like the display on your Xively Workbench. That’s fine for many things, but it is very Web 1.0 so if you want more then take a look at Dave’s method of using Google charts by getting the raw data back from Xively. That way you get a more responsive chart and can plot more than one thing at a time, but you are looking at a lot more work compared to the quick-and-dirty method of an embedded image.

AM2302 (DHT22 ) Temp Humidity sensor and JAL on a 16F628 at 4MHz clock

Having decided I can’t be bothered with digital sensors with oddball serial interfaces like the DHT22 it was time to suck it up when I needed a number of sensors. Cost adds up with lots of sensors – though that Honeywell product more than paid for itself a few times over in much better hatch rates (fertile eggs are about £2 a pop by post, that’s how much of a loss you eat for every failure to hatch!) not every sensor application affects the bottom line like that. Sometimes low cost trumps accuracy, reliability and serviceability. Enter the AM2302, apparently a.k.a. DHT22, produced by the fine Aosong corporation. Their website looks like line noise on my browser, but apparently they are based in Guangzhou, which is China’s third largest city, a conurbation of nearly thirteen million souls.

AM2302, humidity and temperature sensor, a.k.a. DHT22 apparently
AM2302, humidity and temperature sensor, a.k.a. DHT22 apparently

The sensors are cheap, nasty and have poor accuracy, but the price is right, it’s the cheapest way to get a humidity and temperature sensor. Five for £17.70 or a unit price of £3.54 from a Chinese supplier on ebay, Buyincoins ISTR. They have a non-standard one-wire interface. That requires you to be able to tell a 30μS high duration from a 68μS high duration. No problem, eh, even with a PIC running on the internal 4MHz oscillator so each clock cycle is 1μS?

There was already a JAL library for this, called temperature_humidity_dht11.jal so I am in development heaven.

Except it doesn’t work – it acts up after about 20s in the video. It sort of works some times, tantalising short runs of OK in amongst loads of timeout errors. I fiddle with the power supply a little as the AM2302 is claimed to be finicky on the need for 5V. No luck. Tracing the library code I find it barfs around

Continue reading “AM2302 (DHT22 ) Temp Humidity sensor and JAL on a 16F628 at 4MHz clock”

Humidity and Temperature sensor on Cosm IoT

The trouble with geese is that they are waterfowl, ie used to hatching near water, unlike chickens, so you tend to have to control the humidity of the incubator as well as the temperature. Humidity is terribly counterintuitive – unlike temperature humans have no real sense for it. So I was looking at how to make a humidity and temperature sensor.

Goslings. Like most animals, geese don’t look so mean when young

The problem starts with our cheap Chinese import incubator. There’s no real calibration on the temperature dial, and you know it’s a bad sign when the manual tells you ‘when your small poultry is ready’. I was really sceptical about how well this would stabilise the temperature. Quite unfairly as it happens. Continue reading “Humidity and Temperature sensor on Cosm IoT”

Constructing a pyranometer – monitoring Daylight Levels using a photodiode and PIC

One of the things it would be good to track at the farm is light level over the year. It’s part of a tri-sensor I want to develop to read light level, temperature and humidity, which will then radio-report to an OpenKontrol Gateway to log the values centrally.

The correct tool for this job is apparently a pyranometer which uses a thermopile to sense the heat incoming to a black surface. Presumably the thermopile, being a differential device by design, tracks variations in ambient temperature. About 40% of incoming solar energy is visible light, and a similar amount is infrared.

The obvious cheapest tool for the job is a silicon photocell, which doesn’t respond across the whole spectrum, but mainly to visible light and IR. Unfortunately the IR peak coincides with a water vapour absorption peak that makes it sensitive to water vapour too because the spectral response peak is in the IR. David Brooks’ website summarises the issues there. In his other website, he describes the fact that there’s a case to be made for measuring just the visible spectrum for photosynthetically active radiation. I’m going to start with a regular Si photocell and fight that issue later 😉

So there’s a lot wrong with the approach, but as Brooks notes

Despite their shortcomings, solar-cell based pyranometers are widely used for meteorological, environmental, and agricultural monitoring, and their performance relative to thermopile-based pyranometers has been studied extensively. Although they are not suitable for producing stable and highly reliable research-quality data required for detecting small long-term changes in insolation, data from simple solar cell pyranometers are still very interesting. They can be used to characterize the potential of sites to produce solar power and to demonstrate seasonal effects on available solar energy. Pyranometer data recorded at intervals of a minute or so provide a record of cloud cover during the day and it has been shown in peer-reviewed scientific literature [Duchon and O’Malley, 1999] that it is possible to use pyranometer data to classify cloud amounts and types even when individual measurements of insolation are not highly accurate. It follows that such data can also be used to track long-term changes in cloud amount and type – an extremely important indicator of climate change.

Looks good enough for me. I’m going to use a BPX65 because I have some and they’re cheap-ish. The BPW21 is a better choice for visible light only, but it’s dearer, I don’t have one and it’s easy enough to switch the photodiode later on.

Right off the bat I don’t like the design of the system described on David Brooks’ page, because it takes the photodiode and rams the signal into a 470 ohm resistor across the diode. In itself there’s nothing wrong with that, but because the resulting signal voltage is low, to keep the response linear, it demands 12-bit resolution from the datalogger A/D converter though it only uses the few lower bits. Making the sensor cheaper makes the datalogger needlessly dearer without using the extra precision you’re paying for, and makes the whole system more sensitive to noise and crap due to the low signal level. However, it makes it simpler to replicate, which probably was the main design brief for that project, but I can do better because I have control of the whole system.

I’m not going to fiddle about with Arduino here – though Arduino makes some things easier it makes things dearer, and seems to lead people to a penchant for digital sensors like the TSL235R or even truly digitised sensors like the DHT22 humidity sensor with proprietary non-standard interfaces. These get hard to service in future years, compared to an analogue voltage interface.

I want a  single supply transimpedance amplifier that I can limit to 5V so I don’t nuke the A/D converter of my PIC. Fortunately I have one- a CA3140 can run the input down to 0V or even 0.5V lower, can bring the output to 0V, and the strobe input lets me limit the output to 5V if I run the opamp off a higher voltage (haven’t decided there yet – if I run it off 5V then there’s no problem).

Looks like once a minute is the fastest that it’s worth sampling this and light level doesn’t really change that fast. The photocurrent is rated at 10nA/lux, daylight is about 10,000 lux peak corresponding to a photocurrent of 100µA. Given a maximum output of 4V the transimpedance resistor should be about 4/100 MΩ = 40kΩ. 39k is the closest preferred value to that, but I took 47k. This is the UK, I can probably expect a little bit less peak sunlight than noon in the Sahara. With the slow response time needed I can slug the frequency response with the 220nF capacitor – this gives me a 10 second RC time constant.

Transimpedance amplifier
Transimpedance amplifier

You can’t suddenly demand Peak Noonday sun, NOW so I will have to field test this for sensitivity. On the bench it works fine, though the peak output with my bench lamp close to the diode is all of about 400mV. However, it’s not bright as the noonday sun there 😉

The CA3140 is a great opamp for this. Being a MOSfet design, it has virtually no input bias current to speak of, compared to the diode dark current (0.1pA as opposed to 1nA dark current). It can run down to -0.5V on the inputs, and to 0.13V on the output low. I could improve the low end by biasing the input (diode cathode and pin3) up 0.6V on a silicon diode and referring the low end of the A/D converter to that, or just sense it with another input and subtract it in software. The input offset voltage of the opamp is the waek point with MOSFET op-amps – 5mV worst case which would get a bipolar opamp binned.

However, it isn’t the worst limitation on the lowest light that can be sensed, that is the 0.13V saturation point, which would correspond to an current of .13/47k = 2.8µA, an illumination of about 280 lux. This probably needs improving, as it’s higher than a dark overcast day. The 5mV offset voltage limit corresponds to about 10㏓, though it has to be added to the various PIC A/D converter offsets which I haven’t developed yet.

This will need fine-tuning later on, but it’s good enough to get me going and ready to tackle the PIC side of things.


Notes on Silicon Photodiode Detectors, J.D.Riggs, 1983

Setting the Real Time clock Arduino based OpenKontrolGateway

The OpenKontrolGateway a board like an Arduino Uno, but with a SD card interface, three RF board interfaces, an Ethernet board module slot and a DS1307 Real Time Clock with battery backup. It’s a nice collection of all the stuff you need to make a datalogger.

Ciseco OKG
Ciseco OKG

The OKG bundle at £25 gives you everything needed to box it up but I added the RTC kit and SRAM which added £12 to the cost, since a data logger without an RTC is always going to be a pain, where you have to connect up a PC to set it. That’s OK at home but no fun at the farm.

The RTC gave me a hard time right off the bat. There are a few gotchas.

  • You have to set the damn thing,
  • you need to do that to even see it started, since it can need initialisation from the SPI bus before the oscillator will start
  • You need a 2032 coin cell present for it to be able to work too, so no testing that without and thinking you’ll fit that later 😉

So if you ‘scope the crystal and think  ‘ah, that’s why it isn’t working’ then not so fast, you may need to initialise and set it before you see any action there. I found all of this out the hard way. This is what I used to set it, which I largely pinched from LadyAda’s tutorial. One of the nice things about that is it lets you set the clock to the time the sketch was compiled, which save a whole load of faff. What you must not do then is to press reset, or switch off and on again, else it will do that all over again, setting the RTC back to the time it was compiled. You must comment out the line

    // RTC.adjust(DateTime(__DATE__, __TIME__));

and re-upload it. This listing has that line commented out, so you must uncomment it to set the clock, then recomment it.

Continue reading “Setting the Real Time clock Arduino based OpenKontrolGateway”

How to change a LLAP device ID using Teraterm script

Note: after doing this I found you should be able to change the Device ID directly via the serial connection to the device using the AT command ATMY, so I guess that would be ATMYEB here.

You have to send an ack from the control node (usually your PC) within 1/10s of the a–STARTED– message showing up to config a LLAP node. That’s way too fast for me to type. I used Teraterm (the BSD licensed V4.75), which is a SSH and serial terminal program. It also supports scripting, and I used a previous version years and years ago in the old Win95 days.

Here’s a teraterm script to switch ORIGID to DEVID. Start Teraterm connected to your FTDI USB port

Start Teraterm in serial mode on your FTDI cable USB port


Power up a XRF on an XBBO board and connect to it via serial. Once it’s running and in range, ie you see the


message repeated five times when you power up your remote device, power off the device you want to configure. Start the macro,

How to start a Teraterm macro
How to start a Teraterm macro

Here’s the macro

; start of macro
dispstr #$1B"[2J"
dispstr 10 13 "a"ORIGID"ACK------"
send "a"ORIGID"ACK------"
pause 2
dispstr  10 13 "a"ORIGID"CHDEVID"DEVID
dispstr  10 13 "a"ORIGID"REBOOT---"
send "a"ORIGID"REBOOT---"
wait "a"DEVID"STARTED--"
dispstr  10 13 "a"DEVID"STARTED--"
; end of macro

Your device should be set to the new DEVID


Remote sensing and the Internet of Things

Wires. That’s the problem with remote sensing, at least it has been until recently. You needed wire to get the signal back to where you wanted to view it, and often to power your sensor too. That’s a grand PITA. The last time I looked at this, about a decade ago, you could get little RF modules running at around 433MHz but these presented the raw demodulated FM signal. Great for voice but you then needed a modem to wrap around the project. And some sort of protocol stack, possibly.

That exchanged the signal wiring problem for a sensor powering issue, and these radio modules were send or receive so everything would end up fire and forget.

I was chuffed to find there’s been a lot of movement in this field. A lot of it seems Arduino based and I selected PICs when getting into micros, so it is a new learning curve. In researching this I came across JeeLabs and Ciseco. The latter had some £12 bidirectional RF to serial cards, the XRF, which I expected to attach a PIC. However, they seeme ot also have used the microcontroller on the RF board to do some signal conditioning for a few sensors, including temperature via the Dallas 18B20 or a thermistor. Since temperature and battery voltage/contact status are some of the things I want to remote sense that saves me a load of programming grunt-work.

They have also documented a simple serial sensor protocol, LLAP, which fitted my needs. The Internet of Things is all very well but if you need a TCP/IP stack for each battery powered node you need a lot of processing power and electrical power, which is back to wiring again.

So I ordered four XRF boards, a couple of thermistor boards and a XBBO carrier board to interface to an FTDI cable to USB. Assembling the thermistor boards and the XBBO were easy enough, now it was time to test it all out and getting some readings. To do that you have to set your LLAP sensor device to some particular address. and this is where is started getting hard. You have to program them over the air, and you have 100ms to respond to the started command.

Ciseco XBBO board for LLAP devices

That’s great for security, but I don’t type that fast 🙂 Which is why I use this script to do that job.