Dangerous Prototypes Open Source Hardware

OLS preorder 1 fail wall

May 17, 2010, 10:27:55 pm

This post is to catalog OLS bugs, and it looks like there are quite a few. We're just now starting to get a handle on things.

The Logic Sniffer was a comparatively-large first production run of a pretty complicated design. The OLS went through more review and testing than any other OS-hardware project I've worked on. None the less it seems like there's plenty of bugs, and we'll do our best to address them quickly. When scaling from an initial 10 beta prototypes to a full production run there can be lots of corner cases and unexpected problems. As always, this is an open source collaborative effort so your help is greatly appreciated.

Minor fails:
1. The OLS doesn't seem to like (unpowered) USB hubs and some laptop USB ports.
2. Mac users seem to have a problem with the SUMP client, Java, or RXTX.
3. The PIC crystal is out of spec, a better value would be 16MHz. Switching crystals has not solved any defects, so at this time we assume this to be a benign error, but it could be upgraded to a major fail in the future.
4. Some people have intermittent connection problems that stem from the SUMP client software. An updated SUMP with increased timeout seems to help, get it here.

Unknown bugs in shipped hardware:
1. Some bitstreams won't communicate in some hardware. PIC appears to send data, but the FPGA does not respond. We've had interesting results with a probe cable fixing this problem.
2. Sometimes the OLS must be reset multiple times before it will work properly.
3. Some [s:](or all?)[/s:] (a limited number) PICs may have shipped without a bootloader installed. This isn't critical yet as there is no firmware update available.


Unknown bugs in production defects:
1. Lots of defects the FPGA never seems to load itself, PROG_B never goes high. We're waiting on a capture of the traffic on the ROM programming pins to see if the PIC is interfering with the FPGA loading process. Replacing the ROM and FPGA doesn't seem to help.
2. Cannot communicate with the 32bit bitstream. This might be a PIC problem, or a variation of the problem with certain bitstreams not working on certain OLSes.

Current process:
The 'some bitstreams won't communicate' problem is probably related to the production defect #2 as well. We're working on a new bitstream that fixes the issues, as well as implementing a new SPI interface between the PIC and FPGA for non-time based communication. I have a board with this defect and hope to at least determine the component at fault tomorrow.

We're waiting on analysis results of production defects, especially the first one where the FPGA doesn't load.

We're also adding 18F24J50 support to the Bus Pirate PIC programmer so more people can program or reprogram their OLS if it came without a bootloader. We should have a solid update on this by the end of the week. (Update: the programmer erases and IDs the chip, programming is next).

IPenguine has three boards on the way and some pretty advanced test tools (and knowledge ) to help troubleshoot.

Re: OLS preorder 1 fail wall

Reply #1 – May 17, 2010, 11:32:33 pm

[quote author="ian"]Minor fails:
1. The OLS doesn't seem to like (unpowered) USB hubs and some laptop USB ports.[/quote]I have not reviewed the entire OLS schematic, but my first thought was "are there any chips which require 5V?"  In my experience, there are quite a few chips out there which require 5V +/-5%, which is 4.75V to 5.25V.  Since USB could easily provide as little as 4.35V, possibly even 4.01V, then you need a boost regulator for those chips.

At a quick glance, the Eagle files imply that all parts work on 3.3V or less, which seems to rule out any problems with lower voltage being supplied by unpowered hubs or laptops.  If it's not the voltage, then perhaps it's the current or wattage.  Perhaps the regulators themselves are not happy with the USB voltage.  It's also possible that filtering between the USB jack and regulators would be needed to smooth out any transients on the incoming voltage lines.  If you look at the schematics for Microchip's USB PIC boards, you'll see a lot of passive inductors and other parts shielding the USB voltage from the circuit.  The OLS has nothing but a single 10 uF cap, with VUSB directly wired to the regulators. This poses a static discharge risk, but doesn't completely explain the problems you're seeing.

Have you tried measuring the total current draw?  On my USB boards, I always design a jumper so that I can disconnect the USB power and insert an ammeter to measure the current.  I suppose you could also do this with a USB breakout cable that allows current measurement.  If you go beyond 500 mA, or beyond the current specified in your USB descriptors, then a hub or laptop host may simply peter out.

Re: OLS preorder 1 fail wall

Reply #2 – May 21, 2010, 02:21:08 am

I tried uploading 16Kx8 inside, as it would be better for what I was trying to do, but when I do the sump client gives me an error trying to talk to the board.
I even went to Windoze and it sees the com port, but gets an error trying to talk to it.  Went back to 4kx32 and it at least makes lines squigle.  I could not get any meaningful samples, but that may well be my fault.  It is clear it will not talk with the other bitstream installed.

Re: OLS preorder 1 fail wall

Reply #3 – May 22, 2010, 02:18:47 am

I can record, using the default bitstream a few captures, but then it hangs and I have to power cycle it several times before it starts working again?
I have not found a pattern yet, but I am looking for it.    I will try several more cases and see if I can find any consistent behaviors.

Also RLE mode is completely broken on my setup, all channels show lots of transition, even if those pins are grounded.  When I get a capture and I am not in RLE mode the grounded pins are all stable and low.  So it looks like it is working, but the data is random and bad.  Not sure what is up there.

(Please note this, or other of my post are not complaints, just trying to provide feedback.  Thanks for the status updates.

Re: OLS preorder 1 fail wall

Reply #4 – May 22, 2010, 04:35:02 am

rhyde, thank you for reporting every problem you find on/with your OLS. I am sure, actually I know that Ian and Jack appreciate every report of a problem, error or issue with the OLS so no one is happy about those but then feedback, even critical reports are the only way to get down to the source of the problems and get them fixed to help all to hold a working and useful tool in their hands, hopefully rather soon.

My preliminary impression of the pre-order OLS - after having performed visual inspection, basic electrical testing and run a few tests with a 16 channel pattern generator at 20MHz - is this: The hardware design appears to be solid. There are at least two layout/production related issues, the bad soldering/mounting of the LCX16245 buffer chips (solder bridges etc.) and the missing PIC bootloader on many pre-order OLSs. There are issues in the Java Client (that may extend to the PIC firmware) and in the SUMP VHDL code (in particular in the alternative 8bit and 16bit bitstreams and with the RLE option). Fixing and improving the Java client as well as the VHDL code is under way. The main issues that will have to be addressed are a) how to properly test, find and handle problems relating from bad soldering/solder-bridges on the buffer chips and b) how to perform firmware updates on OLSs that have no bootloader should PIC firmware changes become necessary.

Re: OLS preorder 1 fail wall

Reply #5 – May 22, 2010, 08:17:53 am

That matches my assumption, but I am burdened with having not done any hacking in years, that is in hardware, so I am never sure if I have set things up wrong, or if I have a problem.  So far I find that there is some interaction between sump, and serial drivers.  On both the mac and windoze boxes if I unplug the ols, sump will not download again without being restarted.  I get device not found error, even if the device is present from the OSes point of view.

I can do some triggering on the default bitstream and several captures, but lacking another powered hub the above problem interferes with figure out many things.  I still get occasional bad captures of the test environment I have, but I know 4 pins are working.  My eyes nolonger do surface mount parts, and I do not have a scope for such work easily accessible, on then other hand I have had enough interactions with Seeed I am not worried, they will do right by everyone if they are patient and calm.

Just trying to get up to speed and give as much feedback as I can.

Re: OLS preorder 1 fail wall

Reply #6 – May 22, 2010, 10:05:02 am

Not a fail but clearly missing Why is the TRGI, CKI, CKO, TRGO header unpopulated (at least on my preorder 1 OLS units it's missing - it's ok that the ICSP, ROM-ISP and the JTAG headers are not populated on the production units as they are not required for operation)?

rhyde, thanks for the further details. I'd love to get my fingers on your unit because all the units I have (2 prototypes and 3 preorder 1) don't have any communication problems at all. Regarding the buffer chip, I have one OLS with a badly misaligned chip that works just fine (It had one solder spot that may or may not have been critical. Cleaning it was easy). An other board had a solder-bridge I first didn't see but once I had noticed it, it was easy to clean as well.

I know from my own experience how hard visual SMD inspection gets at some point ... but then I have the equipment (stereoscopic microscope etc.) readily available.

On my Windows XP Prof. system I can unplug the OLSs and plug them back in or push reset and I don't have to restart the Java Client nor the SUMP LA.net client. I will get the device not found error only if I don't select the right COM port. Same on my Macbook and Mac mini.

In particular your report about having the same problems on your Mac and Windoze boxes is interesting - could you please answer following questions (you may have already reported the details but I don't remember):

1. Will your OLS work even if only sporadic when you connect it directly to an USB port on your PC/Mac or does it require a powered Hub to work at all?

2. What versions of Windows and Mac OS are you using/have you tried?

3. Are you using the default settings for capture when encountering the communication problems?
  - Port Speed 115200bps(LL)
  - Number Scheme Inside
  - Sampling Clock Internal
  - Sampling Rate 100MHz
  - Channel Groups 1,2,3 and 4 marked
  - Recording Size 2K
  - Noise Filter Enabled
  - RLE Not Enabled
  - Trigger Settings Not Enabled

Regarding Seeeds product support policy I agree that they try real hard to do everyone right if you are fair and a bit patient. At least that's my personal experience.

Re: OLS preorder 1 fail wall

Reply #7 – May 22, 2010, 12:47:53 pm

[quote author="IPenguin"]it's ok that the ICSP, ROM-ISP and the JTAG headers are not populated on the production units as they are not required for operation[/quote]Sorry to semi-hijack this thread, but isn't the ICSP header required to program the PIC Flash with the basic bootloader?  Does Seeed have some other tool for programming the PIC?  If so, I'd like to know, because it could potentially save money on future projects if the ICSP header can be eliminated from the BoM and assembly.

Re: OLS preorder 1 fail wall

Reply #8 – May 22, 2010, 04:19:58 pm

[quote author="rsdio"]
[quote author="IPenguin"]it's ok that the ICSP, ROM-ISP and the JTAG headers are not populated on the production units as they are not required for operation[/quote]Sorry to semi-hijack this thread, but isn't the ICSP header required to program the PIC Flash with the basic bootloader?  Does Seeed have some other tool for programming the PIC?  If so, I'd like to know, because it could potentially save money on future projects if the ICSP header can be eliminated from the BoM and assembly.
[/quote]

There are testing pins, which have some kind of spring inside which keep a pin firmly pressed against a contact. Sparkfun has them and calls them pogo-sticks. They come in various shapes (round, tip, chiseled). Productlink: http://www.sparkfun.com/commerce/produc ... ts_id=9174

They have a tutorial about how they use them: http://www.sparkfun.com/commerce/tutori ... als_id=138

I like the way how they make use of pcbs

But back to the topic: My OLS has finally arrived! It looks like it works ok (haven't looked very much into it yet because i first must make a probe set and testing rig) Is there a simple test(s) to completely test the variuous parts of the OLS?

The progres of making of the probes will be posted in a different topic.

Re: OLS preorder 1 fail wall

Reply #9 – May 22, 2010, 10:11:09 pm

Besides the pogo-sticks Sjaak mentioned there are other adapters and mechanisms to attach external signals to contacts on PCBs without using populated headers/connectors. One very simple example is the Parallax Prop Clip, a FT232 based USB to TTL UART converter that is clipped on contacts at the edge of a PCB board rather than getting plugged into a connector/header - actually it fits right onto the unpopulated contacts of a .100" spaced header. The Prop Clip/Prop Plug documentation shows in detail how the Clip is used.

Sjaak, there is no self-test implemented in the OLS firmware and no test procedure has been published but one test that requires 2 OLS boards. The test is documented and described on the GadgetFactory.net site. However, the links to the test code don't work. You will see a third, green board on the left in the test setup picture. It's a Butterfly USB Cocoon Jack used as a JTAG interface in the test.

I think this project requires a well documented test procedure in the form of a step-by-step guide so users can test their devices and verify if their OLS is working properly, what problems it may have and how they can be fixed. I think this should be discussed in an other thread in detail. 

Re: OLS preorder 1 fail wall

Reply #10 – May 22, 2010, 11:23:10 pm

Answers to your questions.
1. Will your OLS work even if only sporadic when you connect it directly to an USB port on your PC/Mac or does it require a powered Hub to work at all?
On the PC it has always been a powered usb hub, but if you want I could test direct connect.
On the MAC it is directly in to the usb port on the wife's macbook air.


2. What versions of Windows and Mac OS are you using/have you tried?
The Windoze boxes are running XP pro SP 3, and XP media sp3.  One is athlon 64 based, the other intel.
The mac is running 10.6.3 up to date snow leopard

3. Are you using the default settings for capture when encountering the communication problems?
  - Port Speed 115200bps(LL)
  - Number Scheme Inside
  - Sampling Clock Internal
  - Sampling Rate 100MHz
I have used the sample rates from 10k to 100Mhz with equal success.
  - Channel Groups 1,2,3 and 4 marked
Some times only group 1 is checked. 
  - Recording Size 2K
  - Noise Filter Enabled
  - RLE Not Enabled
  - Trigger Settings Not Enabled
Sometimes triggers is enabled and some times it is not.

Some sample files can be found on http://thehydes.net/with.sla  I will be glad to set up and reasonable test sequence or configuraton you want me to try.

Was thinking about building a arduino mega counter that counted all the pins to verify them.  Would that be a useful thing for the group?  I have several megas lying around.

Re: OLS preorder 1 fail wall

Reply #11 – May 22, 2010, 11:32:40 pm

@IPenguin:

Unfortunately I only have 1 OLS here.

I did run some more test using my buspirate. I tied the gnd together and connected AUX to a (buffered) channel. I generated a PWM signal with the BP and checked it with the client. I didn;t have the patience to do all the 16 channels but it did looked well. Need to test all the channels, the presence of the bootloader (pic-part) and the ability to upload new bitstreams.

The command I used on the buspirate (newterm ) was:

Code: [Select]

g 100 25

This would generate a 100KHz waveform with a 25% duty cycle. and is very well to see. it is also a simple check to see if there are no solderbridges.

How about using the unbuffered pins to generate a test pattern? Just a bit like the test with 2 OLS, but only using one. As I guess most people only just have 1 OLS This would at least test the flashrom, the buffer chip and (i guess) most of the electrical connections on the bufferchip/FPGA. This cannot rule out timing issues as the clock is the same within the fpga. Also we nee a seperate test for the PIC and presence of a bootloader.

I dont have the knowledge atm to generate such a bitstream and I must admit I haven't looked thoroughly into this project and might thus be saying stupid things.

I would encourage a topic which helps people with some testscripts and solutions.

Offtopic: that propellor clip just looks like a tiny pogostick Are those available somewhere (i mean the 'contacts' )?

Re: OLS preorder 1 fail wall

Reply #12 – May 22, 2010, 11:48:42 pm

[quote author="rsdio"]isn't the ICSP header required to program the PIC Flash with the basic bootloader?[/quote]

I generally just put a header on the PicKit2 cable, insert it into the holes in the board, and hold it a slight angle with my thumb so it makes contact.  In several years of doing it this way I've never had a problem reading or writing using this method. YMMV of course.

Re: OLS preorder 1 fail wall

Reply #13 – May 23, 2010, 01:02:57 am

[quote author="hank"]
[quote author="rsdio"]isn't the ICSP header required to program the PIC Flash with the basic bootloader?[/quote]

I generally just put a header on the PicKit2 cable, insert it into the holes in the board, and hold it a slight angle with my thumb so it makes contact.  In several years of doing it this way I've never had a problem reading or writing using this method. YMMV of course.
[/quote]Moved to new thread: http://dangerousprototypes.com/forum/in ... opic=573.0

Re: OLS preorder 1 fail wall

Reply #14 – May 23, 2010, 02:46:24 am

Some more testing data.
One the pins seem good.  I wrote a simple counting program and ran it on an arduino then took many dumps are many speeds.  All my pins work, or the 16 buffered ones.  (Goodness for me!)  But I am seeing some USB craziness.
If I connect to the ols via a powered hub, I get a munged device id vid/pid what ever.  Where directly I see usbmodem411 for the name with a hub I see things like usbmodem241121, but the values change when I plug it in?  Most of the time macos thinks it is a network serial device, but the connection is a bit flakey to the device.  Sometime sump will connect to the funny device, and sometime it works, but it is alot less stable.  If I change the port on the hub the device is more likely, but not always, going change.  /dev/tty.usbmodem24131 is another name that worked. 241122 did not work.  No other device mouse arduino, ftdi cables or boards has this problem with the hub.