Dangerous Prototypes

Dangerous Prototypes => Bus Pirate Support => Flashrom => Topic started by: biosflasher on July 30, 2010, 06:39:59 pm

Title: Flashrom extension development
Post by: biosflasher on July 30, 2010, 06:39:59 pm
I couldn't find any docs for binmode (specifically SPI) buffering in the wiki. Do you have any pointers?
I'd like to add flashrom support for this feature because it has the potential to speed up some writes by a factor of 4, maybe more.
Title: Re: Flashrom extension development
Post by: ian on July 30, 2010, 07:14:39 pm
Hey Biosflasher - thanks for posting, I've been looking for you :)

Some reported problems with flashrom and the latest firmware. I tested it but didn't have any problems.

The new bulk mode is for PIC programming, but I want to make you a new SPI command for flashrom too.

How about this:
Two new commands. Long write and long read.
Long write:
Write Command|length H|length L|data0|data1|...|datan|
The long write command, with the length of data to buffer then write out SPI (up to 4096 bytes)
Write all the data at once, max speed. The Bus Pirate buffers it first, then writes it out SPI at max speed.
Bus Pirate responds 0x01 at the end of data transmit.

Long read:
Read Command|length H|length L|
Bus Pirate returns up to 0xffff bytes.

I think that will speed things up a ton :) It speeds up PIC programming from minutes for 18K to 1-2 seconds.
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 03:45:40 am
Hey Ian,

thanks for thinking of me and of the needs flashrom has.

[quote author="ian"]
Some reported problems with flashrom and the latest firmware. I tested it but didn't have any problems.
[/quote]

Do you have a link to reports? I only heard rumors, but never saw any logs.

There is a known problem with having to reset the Bus Pirate after a flashrom run. I have diagnosed that problem and it's essentially a side effect of the "send 0x00 twenty times to enter bitbang mode". flashrom exits binary SPI mode and raw bitbang mode on regular shutdown until the Bus Pirate is in regular user terminal mode again, but flashrom can't do this if it is killed by the user. On the next flashrom run flashrom will send 0x00 twenty times again, but it will get boatloads of "BBIO1" responses instead of just one response. flashrom can deal with that as well by running a "flush serial line buffer" command, BUT flushing actually does not work reliably. Parts of the buffer are flushed by the OS, but since the Bus Pirate is slow and the FT232 is slow as well, the remaining response from the Bus Pirate might arrive after flushing. I tried flushing five times, and it was not sufficient. I tried flushing five times with a 1.5 ms delay between flushes and that fixed maybe 80% of the problems, but in the end I gave up and flushed 10 times with an 1.5 ms delay between flushes and with those changes I have been unable to break initialization.

I'm looking for testers of that patch at http://patchwork.coreboot.org/patch/1673/ (http://patchwork.coreboot.org/patch/1673/) . It works for me, but I have no idea if the rumors about flashrom problems are related or not.

Note: Never trust a flush command to actually do what you want.

[quote author="ian"]
The new bulk mode is for PIC programming, but I want to make you a new SPI command for flashrom too.

How about this:
Two new commands. Long write and long read.
Long write:
Write Command|length H|length L|data0|data1|...|datan|
The long write command, with the length of data to buffer then write out SPI (up to 4096 bytes)
Write all the data at once, max speed. The Bus Pirate buffers it first, then writes it out SPI at max speed.
Bus Pirate responds 0x01 at the end of data transmit.

Long read:
Read Command|length H|length L|
Bus Pirate returns up to 0xffff bytes.

I think that will speed things up a ton :) It speeds up PIC programming from minutes for 18K to 1-2 seconds.
[/quote]

A few days ago I committed a patch to flashrom which resulted in a 3x speedup on the Bus Pirate by violating the protocol a bit. It essentially bundles three commands and sends them off before waiting for a response for the first command. Those three commands are: Lower CS#, Read/Write SPI, Raise CS#.

How can flashrom detect the availability of your proposed new mode? I want to make sure that this feature needs zero user interaction (autodetect is needed) and won't break old flashrom versions. Some people won't be able to upgrade their flashrom version, and others won't be able to upgrade their Bus Pirate firmware. We should take care of both groups.

Now on to the technical details:
For writes it is essential that the buffer is big enough to actually write in one go without stopping the clock in between. While most chips can handle a stopped clock just fine, other chips explicitly require that all clock line states (1 or 0) are not longer than 50 us (lowest allowed frequency is 10 kHz). Due to that, flashrom has to take the buffer size of the Bus Pirate into consideration.
The trick of bundling up commands has to continue working or the new feature will essentially make flashing slower than it is right now, and that would be really sad.
However, there is one additional 2x speedup on some writes we could get by being able to bundle six commands: Lower CS#, Read/Write SPI, Raise CS#, Lower CS#, Read/Write SPI, Raise CS#. Most SPI flash write commands are actually two SPI commands in direct succession, and flashrom is able to send them at once if the hardware supports that.
Your proposed SPI read/write commands look good at a first glance. Now if there is a way to make sure the delay between subsequent write and read commands is less than 50 us, this should work out nicely, and we get the advantage of reducing traffic over the serial line by half.

Oh, and while I'm outlining a wishlist there are two things which would result in a ~100x speedup for some commands:
A repeat-until-result-mask command which would mask+compare the SPI response for the previous command against some value and repeat until response&mask=value. Usage would be to send read-status-register to the chip and repeat that until the response from the chip has the work-in-progress bit cleared. Having the ability to limit the number of loops even if the condition is not met would be even better since flashrom doesn't want to be stuck on a dead chip without working diagnostics.
As an alternative (or useful extension), a delay-n-microseconds command for that mode would help as well. Many chips offer a choice between waiting for a fixed time or polling the status register to detect end-of-write.
Either solution would be faster than a full flashrom<->Bus Pirate roundtrip to check the status register.

There is another speed problem I'm having with the Bus Pirate, and I was unable to track down the underlying reason. Communication with the BP is roughly 9x slower than it should be. AFAICS it should be possible to use one USB frame to send a command to the BP, and use the next USB frame to read the response. That would give us a roundtrip time of 2 ms per command (1 ms per USB frame). However, the actual roundtrip time is closer to 18 ms which is really devastating for performance. I've seen reports which indicate the problem exists on Windows and Linux, and it apparently is independent of how much data we're reading or writing from/to the BP. Just to give you an idea of the magnitude of problems: MrHijet needed 10 hours to write 1 MB, and that is totally unacceptable if we could manage this in little more than an hour without my speedups, and maybe 20 minutes or less with all possible speedups for the old BP SPI interface.

Do you have any idea what could cause those massive roundtrip delays?
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 10:10:12 am
Here's a few posts that I replied to when I double checked everything in v5.2:
http://dangerousprototypes.com/forum/in ... 22#msg6522 (http://dangerousprototypes.com/forum/index.php?topic=154.msg6522#msg6522)
http://dangerousprototypes.com/forum/in ... 70#msg5970 (http://dangerousprototypes.com/forum/index.php?topic=651.msg5970#msg5970)

The binmode replies 0x00 when a command is unknown, and 0x01 when it is known, that is probably the most straight forward way to test.

I'm not sure about the round trip delays, there is a setting in the FTDI driver that determines how frequently it polls for new data. The delay is long enough that programs that use timeouts (SUMP logic analyzer mode, for example) will frequently timeout waiting for the next packet.You can change it under the control pannel in windows.

I'm not sure what to do about bundling. I'd be happy to add extensions, or even a whole flashrom protocol, but we're nearly out of room in the current chip/firmware ( a few hundred words left). When Bus Pirate v4 comes out we could revisit this again because there will be 4x more space (and an internal USB connection for much faster coms).

For now, I've attached a firmware with three new commands to hopefully go some distance towards removing some of the bottleneck and bulk when doing long data transfers.

               case 0b0100: //long read
               case 0b0101: //long write
               case 0b0110: //long pump
about here in the code:
http://code.google.com/p/the-bus-pirate ... /SPI.c#594 (http://code.google.com/p/the-bus-pirate/source/browse/trunk/source/SPI.c#594)

the format for all commands is the same:
command|high 8 length|low 8 length|

Read will then return up to 0xffff bytes as fast as SPI will allow.

Write will buffer up to 4096 bytes and then write them all to SPI as fast as it will allow. At the end of the transfer it returns 0x01. If you ask for more than 4096 bytes in this mode it returns 0x00.

Pump does essentially what you're already doing, but has a check for overflow. Pump moves up to 0xffff bytes from the serial port and puts them on SPI. As long as the SPI can keep up with the UART (it is 8MHz? has 4 byte buffer, probably ok) it will moves bytes as fast as it can. If the SPI buffer fills at any time it stops and sends 0x00.

We could add a configuration option to raise and lower CS automagically for each of these new commands.

Just a note - with the new servo mode and these changes the firmware is too big. I removed the easter egg, but that only gives a few hundred extra words.
Title: Re: Flashrom extension development
Post by: Sjaak on August 02, 2010, 10:56:27 am
Not the easteregg!

/me cries

Would it be time to adjust the ID string with a higher version number? Please also make the CS selectable (active high/low)
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 11:09:13 am
I think we could reclaim a bunch of space if we moved the binary mode to v2 and implemented a consistent interface that doesn't need to have all the stuff repeated for each protocol. I started something like this (new bin) but with all the libraries having different commands it was a lot of trouble and didn't save any space.
Title: Re: Flashrom extension development
Post by: Sjaak on August 02, 2010, 12:04:27 pm
It has become trashbin? :D

All the protocols have standard entrypoints for busstart, stop, read , etc. Take a look at the scripting language how to 'binairy' interact with the protocols.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 12:13:37 pm
It is without a doubt trashbin. It's the binmode protocol that is all over the place. Some of the commands are the same across all binmodes, but some have sniffers, some have bitwise where others have bulk reads, etc.
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 03:46:25 pm
Hey Ian, thanks a lot for implementing Long Read and Long Write. They look pretty useful to me.

To be honest, I have absolutely no idea what the pump mode does, even after staring at your description and the firmware. A more detailed explanation would be appreciated. Or is the pump mode just a longer variant of the old Read/Write command?

With your new commands, flashrom will get a 2x-21x speedup for reads (UART speed is the limiting factor here), up to a 21x speedup on write for some chips (those supporting page write) and no speedup on write for other chips (those supporting Byte/AAI write).

The new command sequence will be
Lower CS#, Write Long, Read Long, Raise CS# instead of the old
Lower CS#, Write/Read SPI, Raise CS#.
flashrom needs a guarantee that the delay between Write Long and Read Long is not longer than 0.05 ms under any circumstances (e.g. even if the OS is stalling writes over the UART). If that is not possible, flashrom can't use Read Long and will have to fall back to the old method.

Automatic control of CS# looks like a great way to save two Bus Pirate commands per SPI command and would remove the need for the current 3x speedup hack because that 3x speedup would be native.

What happens if flashrom aborts during a Write Long (4096) command because the user killed flashrom? AFAICS on the next flashrom run init will hang because it only sends 20 bytes of 0x00 and waits for a response that won't happen because the Bus Pirate thinks this is still input for the Write Long. Worst case is that flashrom init will hang 205 times before init starts working again. A possible workaround is to send 4096 bytes of 0x00 on init, but that might cause significant startup delays. Would a timeout for Write Long make more sense?

Side note: If firmware space is really that tight, just tell me what the minimum size of the command/UART buffer is, and I will simply bundle up commands until the limit is reached (if the Bus Pirate can handle that). That would probably result in almost the same read speedup (though now write speedup) as the new commands, but without the need for new commands.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 04:03:29 pm
Pump is an alternate write method, instead of storing all the data in a buffer then flushing it out SPI, it just takes from UART and puts it in SPI. This is fine for 10MHZ spi, but at low speeds the SPI buffer (4) will fill and we start to loose data.

Yes, it will just be stuck waiting for the bytes to finish. A timeout could be added eventually.

It seems simple enough to just give you a single flashrom extension, it will take less than the long stuff maybe.

How about this:
command|write H | write L| read H | read L| write data... | read data....

I know we've had this discussion before, but I can't find it... The most common page size is 256bytes as I recall, but many chips can be continuously read from 0 to .... mbits. The OLS flash is 264bytes/page (or something), so it seems to make sense to have more than 256bytes as a write/read minimum, but then one-byte read/write will have a ton of overhead. Are all pages /8 or something? Then we could use a single byte to represent chunks of 256 bytes to read.

This function would have automatic CS. I'll implement it now.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 04:24:09 pm
I split this into it's own topic and made a new board for flashrom. It's under BP development right now, but it should probably go up front with the OpenOCD support.

Attached in a new firmware with a command in that format. To ensure that it meets the most stringent timing requirements is gives up some speed and buffers everything:
1. Send any of the old long commands
2. send H/L number of bytes to write (max 4096)
3. send h/l number of bytes to read (max 4096)
4. BP replies 0x01 ok or 0x00 error (too many bytes)
5. Send the write bytes. they are buffered in the Bus Pirate.
6. CS is low, all write bytes are sent at once
7. read starts immediately, all bytes are put into a buffer at max SPI speed (no waiting for UART)
8. ENd of data CS goes high
9. buffered read bytes are returned.

We can optimize how represent the number of read/write bytes for the least overhead and maximum flexibility.
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 05:15:48 pm
[quote author="ian"]
I split this into it's own topic and made a new board for flashrom. It's under BP development right now, but it should probably go up front with the OpenOCD support.
[/quote]
Thanks!

[quote author="ian"]
Attached in a new firmware with a command in that format. To ensure that it meets the most stringent timing requirements is gives up some speed and buffers everything:
1. Send any of the old long commands
2. send H/L number of bytes to write (max 4096)
3. send h/l number of bytes to read (max 4096)
4. BP replies 0x01 ok or 0x00 error (too many bytes)
5. Send the write bytes. they are buffered in the Bus Pirate.
6. CS is low, all write bytes are sent at once
7. read starts immediately, all bytes are put into a buffer at max SPI speed (no waiting for UART)
8. ENd of data CS goes high
9. buffered read bytes are returned.
[/quote]
Having dedicated commands for flashrom in the firmware is awesome!
One of the biggest impacts on read/write speed is round trip time. Anything which reduces round trip time is very desirable. One way to avoid the roundtrip at point 4 would be to have a command which queries the buffer size and have flashrom avoid overflows automatically. This would allow us to move point 4 directly before point 9. AFAICS this is worth a 2x speedup.

What do you mean with "old long commands"? The new command you're outlining looks like we should call it flashrom_writeread_spi or something like that, and it would be the only thing flashrom needs.

Side note: The auto-CS# handling might make sense to happen for the really old SPI read/write interface as well, but I'm not sure if there is a point to that. It might have lower overhead for really short writes (no 4 bytes for length needed), but that's pretty much it.

[quote author="ian"]
We can optimize how represent the number of read/write bytes for the least overhead and maximum flexibility.
[/quote]
Good point. For starters, flashrom will always write at least 1 byte. This allows us to cram the write size in 12 bits. If we restrict the new command to a read size of 4095 bytes, we can stuff the read size in 12 bits as well. That saves one byte for transmission, but I have no idea if it makes the firmware bigger or smaller. The timing impact is probably negligible for most operations.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 05:27:26 pm
Quote
What do you mean with "old long commands"?

 It responds to the same command. I just stuck this new command under the same switch as the new commands in the last build.

Quote
One way to avoid the roundtrip at point 4 ...

Sorry, that line was accidentally left there when I compiled... It wasn't intended (jsut used it for testing), but in case you tried it I wanted to give you a heads up. I attached a new version without it.

I figure we still need the 0 still if someone wants to do all read or all write in one operation.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 05:29:42 pm
1. Send any of the old long commands
2. send H/L number of bytes to write (max 4096)
3. send h/l number of bytes to read (max 4096)
4. Send the write bytes. they are buffered in the Bus Pirate.
5. CS is low, all write bytes are sent at once
6. read starts immediately, all bytes are put into a buffer at max SPI speed (no waiting for UART)
7. ENd of data CS goes high
8. buffered read bytes are returned.
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 06:47:28 pm
The interface looks really good, and the code is very readable. Thanks!

I'd kill all long commands except one. There is no software out there using them, and they take up valuable space in the firmware. Just tell me which one to use and I'll stick to that.

A getbufsize command (returning OK, hi, lo) would be really appreciated because it has twofold functionality:
For most writes, flashrom will send two commands (WRITE_ENABLE and PAGE_WRITE) directly after each other (of course CS# is disabled and enabled in between). WRITE_ENABLE writes 1 byte to SPI and reads 0 bytes. The subsequent PAGE_WRITE writes 5-260 bytes (1 command, 3 address, 1-256 data for classic NOR flash with 256 byte sectors). Can I just send both commands in the same write() to the serial port and wait for their result after both have been sent, or is that a problem with the UART/BP buffer? Such command bundling would result in another 2x speedup.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 07:05:52 pm
The UART buffer has 3 bytes. The first is command and number of bytes, which is processed right away, then one byte to send, then CS, then setup long and send... I think you can bundle that because the only bottleneck is the one byte sent over SPI, as long as 3 more bytes don't arrive over the UART in the time it takes to send that one SPI byte it should be OK.
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 07:07:23 pm
I just noticed that r461 removed UART1TX(1). Was this intentional or did you just want to move it down a bit? Moving it down might have speed advantages, but removing it completely is a problem if flashrom wants to differentiate between an error return and the response from a flash chip.

Mental note to self: flashrom expects to read the full length of a success response even in the error case. Fix that to avoid eternal delays in the error case.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 07:14:15 pm
Yes, I removed it because I thought it would speed things up, and other commands that return reads don't ack with a 0x01, but it is needed on the no read write or it looks like nothing happened. I've added it back in and recompiled. You can get the latest version from SVN here:

http://the-bus-pirate.googlecode.com/sv ... e-v5.5.hex (http://the-bus-pirate.googlecode.com/svn/trunk/firmware/v5-nightly/BPv3&v2go/BPv3-Firmware-v5.5.hex)
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 07:28:44 pm
[quote author="ian"]
The UART buffer has 3 bytes.
[/quote]
Ouch. Is that the buffer of the Bus Pirate or the buffer of the FT232 chip?
The FT232R documentation suggests that it has "256 Byte receive buffer and 128 Byte transmit buffer".

[quote author="ian"]
I think you can bundle that [...].
[/quote]
Thanks for confirming.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 07:51:24 pm
That is the hardware buffer of the PIC UART for incoming and outgoing bytes (3 each). It's actually pretty good, most uCs have 1 bytes (none) only.

We have another software buffer in the PIC. The FTDI chip has it's own buffer too.
Title: Re: Flashrom extension development
Post by: biosflasher on August 02, 2010, 08:00:36 pm
[quote author="ian"]
Yes, I removed it because I thought it would speed things up, and other commands that return reads don't ack with a 0x01,
Strange. In my copy of the firmware, the 0b0001xxxx command (SPI read/write) does return an ack. I'll have to check if my copy is corrupt.

[quote author="ian"]
but it is needed on the no read write or it looks like nothing happened. I've added it back in and recompiled. You can get the latest version from SVN here:
[/quote][/quote]
Thanks!

I am very sorry that somehow it seems we don't understand each other in the ack case. AFAICS the problem I'm trying to tell you about was created by r461, and r462 doesn't fix it. I'll try to explain.

flashrom sends 0b00000100 writelen.hi=0 writelen.lo=1 readlen.hi=0 readlen.lo=1 one_byte_of_data=READ_STATUS_REGISTER

In the error case (writelen/readlen too long),
Bus Pirate returns 0x00

The flash chip responds with 0x00 to the READ_STATUS_REGISTER command.

In the success case with r461,
Bus Pirate returns 0x00 and there is no way to differentiate between an error and the response from the flash chip which just happens to be zero for that command.

In the success case with r462,
Bus Pirate returns 0x00 0x01 and there is no way to find out if the first byte of the response (0x00) is an error code or the first byte of data, so flashrom has no idea whether it should continue reading.

AFAICS the Ack would be best placed directly before the loop which transmits buffer contents over the UART. Another thing is CS# timing with regards to the UART. May I suggest a slightly modified code flow? I moved SPICS changes directly before the write-from-buf and directly after the read-to-buf. Besides that, the Ack response is now sent before the buffer is returned, but after all the timing critical stuff is done. This means that once the Ack is in, flashrom knows that the write/read operation is over, and all that remains is fetching the response from the chip. I also killed case 0b0101 and case 0b0110.
Code: [Select]
                                        case 0b0100: //write-then-read
                                                //get the number of commands that will follow
                                                while(U1STAbits.URXDA == 0);//wait for a byte
                                                fw=U1RXREG; //get byte
                                                fw=fw<<8;
                                                while(U1STAbits.URXDA == 0);//wait for a byte
                                                fw|=U1RXREG; //get byte

                                                //get the number of reads to do
                                                while(U1STAbits.URXDA == 0);//wait for a byte
                                                fr=U1RXREG; //get byte
                                                fr=fr<<8;
                                                while(U1STAbits.URXDA == 0);//wait for a byte
                                                fr|=U1RXREG; //get byte


                                                //check length and report error
                                                if(fw>=TERMINAL_BUFFER||fr>=TERMINAL_BUFFER){
                                                        UART1TX(0);
                                                        break;
                                                }

                                                //get bytes
                                                for(j=0; j<fw; j++){
                                                        while(U1STAbits.URXDA == 0);//wait for a byte
                                                        bpConfig.terminalInput[j]=U1RXREG;
                                                }
                                              
                                                SPICS=0;
                                                for(j=0; j<fw; j++){
                                                        spiWriteByte(bpConfig.terminalInput[j]);
                                                }

                                                for(j=0; j<fr; j++){ //read bulk bytes from SPI
                                                        bpConfig.terminalInput[j]=spiWriteByte(0xff);
                                                }
                                                SPICS=1;

                                                UART1TX(1);//send 1/OK

                                                for(j=0; j<fr; j++){ //send the read buffer contents over serial
                                                        UART1TX(bpConfig.terminalInput[j]);
                                                }

                                                break;

If I'm seeing problems where there are none, please tell me.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 08:27:24 pm
Sounds great. I replaced the routine and uploaded a new nightly compile. Save URL as my previous post.
Title: Re: Flashrom extension development
Post by: ian on August 02, 2010, 08:30:44 pm
You're right about the ack to 0xa0 + bytes, it does ACK in SPI. I forgot about that. The new PIC modes don't do that, I'm trying to make everything a little faster.
Title: Re: Flashrom extension development
Post by: ian on August 04, 2010, 07:42:10 pm
Have you had a chance to test the new command? Just let me know if there's anything I can tweak for you.
Title: Re: Flashrom extension development
Post by: biosflasher on August 05, 2010, 03:47:01 pm
Not yet. The problem is that if I upgrade my Bus Pirate, I don't have a way to test new flashrom against old Bus Pirate firmware anymore. From a support perspective, that would be unwise since people expect flashrom to continue working just fine even if their Bus Pirate is at an ancient revision.
I could post my patch to the flashrom mailing list and take you in CC so you (or anyone else interested) can test. I also know a few people who own a Bus Pirate and might be interested in testing a new firmware together with the patch.
What do you think?
Title: Re: Flashrom extension development
Post by: Sjaak on August 05, 2010, 04:12:15 pm
If you have a pickit or icd (or any pic24 programmer) you could always revert to an older version (bootloader). If you don't have any we could write a bootloader v2 installer for you.
Title: Re: Flashrom extension development
Post by: ian on August 05, 2010, 04:14:20 pm
What version are you currently at? You can install any v4+ firmware with the v4+ bootloader, going back to v3.6 or earlier is a problem.

To make a universal version, the first time you send the new flashrom command, do a short pause, and then check for 0x00 (unknown command). If you get 0x00, the command is unknown, fall back to the old method and remind the user to upgrade for better speed.

I'm curious about the speed increase. Lots of people have complained about really long read/write times, I think this extension will help a ton. When I went from short packets to bulk reads in the PIC programming support, reads went from an hour to seconds.
Title: Re: Flashrom extension development
Post by: biosflasher on August 06, 2010, 02:23:39 am
I actually tried autodetection with the old firmware and it had spectacular side effects. You see, checking for availability of a command is nice in theory, but in practice it goes like this:

Send 0x04 0x00 0x00 0x00 0x00 (newSPI, writelength 0, readlength 0)
The new Bus Pirate responds with 0x01 (Ack). Good.
The old Bus Pirate responds with 0x00 0x42 0x42 0x49 0x4f 0x31 0x42 0x42 0x49 0x4f 0x31 0x42 0x42 0x49 0x4f 0x31 0x42 0x42 0x49 0x4f 0x31  (Nack, BBIO1, BBIO1, BBIO1, BBIO1). Slightly inconvenient. This also means that if I try anything besides writelength=0, readlength=0 the bitbang mode will interpret them as commands and end up in some random (well, dependent on writelength/readlength) mode and random code execution in that mode.

Just sending the first byte of the new command and looking for Ack/Nack won't work either: The Nack (old firmware) comes immediatedly (good). The Ack (new firmware) is only sent after writelength and readlength have been shifted in, so I can either wait a predefined time (which is unreliably by definition) for a Nack before I decide to continue, or I simply hope that a stream of 0x00 in SPI mode will always (for all future generations) exit SPI mode and end up in BBIO mode.
Title: Re: Flashrom extension development
Post by: Sjaak on August 06, 2010, 07:45:25 am
how about changing the modestring (SPI1) to SPI2?

No more secrets handshakes with uncertain outcomes... ;)
Title: Re: Flashrom extension development
Post by: ian on August 06, 2010, 09:40:06 am
Quote
so I can either wait a predefined time (which is unreliably by definition) for a Nack before I decide to continue, or I simply hope that a stream of 0x00 in SPI mode will always (for all future generations) exit SPI mode and end up in BBIO mode.

No worries then, I guessed you already had a function with a timeout for checking for ack/nack that you were comfortable with. The Bus Pirate does return the NACK 'immediately', as in there's no bus operations that delay it, but I get that that's not predictable enough.

I'm hesitant to change the version string because other apps (AVRdude) would have to be updated even though their commands were not effected. If mode is SPI1, 0x00 will always exit SPI and end up in BBIO.

If this is the only thing in the way of a speedier protocol, I'm willing to add an alternate command that simply replies 0x01, but will fail 0x00 in older versions. Or we could add something that prints the version string in BBIO mode, then you could judge which protocols are supported.
Title: Re: Flashrom extension development
Post by: Sjaak on August 06, 2010, 12:33:16 pm
[quote author="ian"]
I'm hesitant to change the version string because other apps (AVRdude) would have to be updated even though their commands were not effected. If mode is SPI1, 0x00 will always exit SPI and end up in BBIO.
[/quote]

I can see your point :) I think it is the only safe way to tell if the improved commandset is there. Alternatively before entering the binmode see what the output is of command 'i' (not as neat as a new version number.
Title: Re: Flashrom extension development
Post by: ian on August 06, 2010, 01:54:43 pm
It's a bit hackey, but the easiest thing that will probably take the least time and program space is to add command 0x06 to SPI that returns the flashrom protocol version number. On old firmware it will return 0x00, in this version we can return 0x01, if it changes we can return higher numbers.
Title: Re: Flashrom extension development
Post by: biosflasher on August 07, 2010, 04:25:40 am
Implementing a reliable(!) cross-platform receive-with-timeout that actually works would be an interesting challenge.

I know that the Nack reply was also meant as a way to check for the presence of a command. May I suggest an alternative which would be generic and take care of this, yet stay backwards compatible?
Add a new check-if-command-available (CHECK) command. The command could be 0xff to make it obvious that it is special...

To use it, you send
CHECK CHECK.
Old Bus Pirates will respond with Nack Nack
New Bus Pirates will respond with Ack

To check for the availability of a command once you know CHECK is available, send
CHECK WHATEVERCOMMAND
If WHATEVERCOMMAND is supported, new Bus Pirates will send Ack
If WHATEVERCOMMAND is not supported, new Bus Pirates will send Nack
Old Bus Pirates should never be called with CHECK+WHATEVERCOMMAND except CHECK+CHECK

The scheme will break down if we ever get multibyte commands where the first byte alone is not enough to check if the command is supported. OTOH, I didn't hear of anyone planning such commands.

Thoughts?
Title: Re: Flashrom extension development
Post by: Sjaak on August 07, 2010, 10:06:41 am
I would say ditch the backwards compatibility! The programs that use it should check if _minimal_ SPI1 is returned, and also accept SPI2 and higher.

The version string is ment IMHO to tell the current commandset, i.e. SPI1 is the old one and SPI2 is the newer one.

If we don't want to go this route, flashrom should issue the command 'i' and determine the firmware and check if this a minimal release. Alternatively an binary version of the command 'i' could be implemented. Spacewise I would then opt for the 'i' command.

the 'i' command: http://dangerousprototypes.com/docs/Bus ... nformation (http://dangerousprototypes.com/docs/Bus_Pirate_menu_options_guide#I_Hardware.2C_firmware.2C_microcontroller_version_information)
Title: Re: Flashrom extension development
Post by: biosflasher on August 07, 2010, 03:34:59 pm
Checking for SPI1/SPI2 is a good idea, but this means every flashrom user would have to upgrade flashrom to use a newer firmware at all, even if the newer firmware still supports the old commands.

How future-proof is it to enter bitbang mode, exit it, run "i", parse the result, and enter bitbang mode again? Will modular Bus Pirate firmware (only a selected number of features) break such detection?

Or do we simply take the easy route and hack around the issue for now?

Another option (which may or may not be feasible anymore) is to use feature bits. One bitfield for backwards incompatible changes, and one bitfield for new features which don't break backwards compat... something similar has been used for the ext2/ext3/ext4 filesystems and it worked pretty well in that case, even though there were countless extensions.
Title: Re: Flashrom extension development
Post by: ian on August 07, 2010, 04:30:14 pm
I have committed to keeping the info screen the same for scripts, though a version command would probably be best. I'll try to add one on Monday.
Title: Re: Flashrom extension development
Post by: Sjaak on August 07, 2010, 05:23:19 pm
[quote author="biosflasher"]
Checking for SPI1/SPI2 is a good idea, but this means every flashrom user would have to upgrade flashrom to use a newer firmware at all, even if the newer firmware still supports the old commands.
[/quote]
Basicly yes. But you could say when we change it now the features are now mature and working. version 5.6+ is the way to go for fast programming and the older versions only slow programming is reliable. Changing firmwares is btw easier then it sounds. With the v4 bootloader you could go back to firmware 4.0. If you (or the adience) really needs it, we could either recompile the old sources or  write a bootloader v2 installer. (don't think it make sense, since most of the feature from the past are still in). Everyone should upgrade in my opinion.

[quote author="biosflasher"]
How future-proof is it to enter bitbang mode, exit it, run "i", parse the result, and enter bitbang mode again? Will modular Bus Pirate firmware (only a selected number of features) break such detection?
[/quote]

I would suggest to keep a minimum of sump, binmode, 2wire, 3wire with the scripting/usermacros in (the base). Then spread the protocols and make the build. The info presented by 'i' would off course tell which things are compiled in (or an other command 'I' perhaps).

[quote author="biosflasher"]
Or do we simply take the easy route and hack around the issue for now?
[/quote]

IMO no. Then it will be a hack forever.. (it will never addressed again)
Title: Re: Flashrom extension development
Post by: rsdio on August 08, 2010, 03:13:52 am
In order to distinguish versions, can you program the Internal EEPROM in the FT232R so that it has a different Device Version Number?  The VID/PID would have to stay the same, but maybe the version could change.