Open Hardware Summit 2012: USB and Open Source presentation
Giving my talk on the problems of using USB in Open Source projects at OHS.
You can download the latest files (Open Office Presenter, or PDF) from our SVN.
This entry was posted in conferences and tagged 2012 Open Hardware Summit.
Comments
The vendor ID problem is made far worse than it needs to be, by the common practice within the open hardware community of using the VID/PID pair to match drivers.
For example, if you use an Arduino Uno (on windows), you need to load Arduino INF file. Then you use a bus pirate, and load another INF file. Copying an INF file and just modifying the VID/PID numbers is easy for the developer, but terrible for the larger community. It doesn’t need to be this way.
If you have ever used my Teensy board, my INF matches against the protocol numbers, NOT the vendor ID. So if you then plug in any CDC-based Arduino, or any other CDC-ACM protocol device (which are so common among open hardware projects), that same INF file will load Microsoft’s driver.
The USB-IF is probably not going to abandon their primary source of sustaining revenue. But the open hardware community can change its practices to dramatically lessen the problem. I’d suggest 2 important things were missing from your presentation:
1: Advocacy for designing devices to standards, and writing INF files (and drivers, in the rare cases where projects actually create a driver) to ignore vendor/product ID numbers and match to the class/subclass/protocol numbers. If every open hardware project using CDC wrote then INF files as I did for Teensy, everybody would only suffer the unsigned driver install just once, rather than going through it over and over again.
2: Recognition that much of USB’s value comes from the standardization process. Sadly, many people in this community will use offensive language like “greeyd bastards” (as seen in previous comments on this site) and frame any issue in unconstructive “us vs them” terms. It’s all to easy to take good things for granted. Any responsible presentation on this issue should recognize the USB-IF’s important role in creating much of the value USB currently has, hopefully to avoid propagating negative sentiment.
Is there a video recording of this talk available?
I think there was a stream of the whole OHS, maybe it is on youtube or the OHS site? We recorded it and I had my personal audio mic on for our geek tour video so we will have a copy too.
Paul,
Your argument depends entirely on 1) using Windows, and 2) using a standard USB class. But it doesn’t apply at all on an OS-agnostic product that uses a custom protocol. And using the serial port emulation over USB has got to qualify as the slowest, silliest use of USB.
USB is great for making simple but structured communication protocols. Why would you restrict yourself to an outdated semi-standard with all the associated historical cruft — baudrates, framing bits, no multiplexing facility at all, etc etc.
Like it or not, when you want to do the job right, you need a custom VID:PID to uniquely identify your device to the OS. And while nobody ever disputed the fact that the USB-IF does valuable work, there’s also no disputing the fact that they’re selling single numbers out of a namespace of 65K at $2000 a pop; that is excessive by anyone’s standards.
Ian,
Congratulations on finally getting that presentation made!
this was a great talk :)
Bert,
Take a look around the open hardware world. A huge portion of all the projects running on USB-enabled microcontrollers are indeed using CDC-ACM protocol.
Your opinion, that anything serial is “outdated” is shared by some others, particularly Microsoft (that’s the reason they have decided not to include an INF that automatically loads their own driver…. which I have heard from a contact within Microsoft trying to get the Windows 8 team to change their position on this INF). That’s your opinion. But when it comes to the technical facts, you’re simply incorrect regarding numerous technical matters.
CDC-ACM doesn’t use baud rates and framing bits for its communication. The data is merely transported as USB bulk protocol, allowing nearly all the available bandwidth to be used. You may believe it’s “the slowest, silliest use of USB”, but CDC-ACM is indeed perfectly capable of using the entire available USB bandwidth. For an example, see the LED Video Wall (6144 RGB LEDs at video speed update) project at dorkbotpdx.org, which uses an open source example I published where CDC-ACM can stream at full USB speed. If you watch the brief video with the band playing in front of the display, that entire display is streamed through a single USB port running CDC-ACM protocol converted to 8 SPI ports! Another example is the Domestar project, which also connected thousands of LEDs and claimed to be capable of a 240 Hz refresh rate. I believe they accomplished that feat using 5 USB ports, each running CDC-ACM.
CDC-ACM really does provide easy access to the full USB speed, as those impressive projects absolutely demonstrate. Serial parameters like baud rates and data framing and serial handshaking signals are merely parameters passed in control messages.
If CDC-ACM has “no multiplexing facility at all”, how would you explain Teensy and Arduino Leonardo implementing a Serial + Keyboard + Mouse device? Dean Camera’s LUFA library provides examples that implement 2 separate CDC-ACM ports on a single chip, and other examples combining CDC-ACM serial with other protocols. TI’s launchpad, demos from Freescale, and others also multiplex CDC-ACM quite well with proprietary protocols on other endpoints. These are opinions, but solid technical facts backed up by working implementations.
But in the context of this issue, that VID assignment is an obstacle to open hardware projects, I couldn’t diagree more with your opinion “when you want to do the job right, you need a custom VID:PID to uniquely identify your device to the OS”. That is *exactly* the sort of conventional thinking which makes this problem far worst than it needs to be.
The primary characteristic open source projects is the sharing of technical design. Sharind any design that requires unique a VID-PID identifier makes this problem far worse. For a custom protocol using libusb or a native driver, recognizing the intended device by VID-PID in host implementaton is the easy and conventional way, but when it comes to facilitating sharing, publishing code that depends upon a VID-PID pair, or a list of such identifiers, is the counterproductive to sharing.
There are lots of alternatives. For a fully custom protocol, you can use 255 for a “vendor defined class” and add your own vendor-defined descriptors. Yes, it is extra work on the host-side implementation to parse this stuff. Using only the VID-PID pair is easier. But in the context of sharing designs in a community, that extra work has clear benefits.
Sure, you still need to put some numbers into the VID and PID bytes of the device descriptor. But the problem is much less important when the host side implementation ignore those fields and looks for other characteristics in the descriptors. More importantly, for the larger open hardware community, building upon widely shared code which ignores VID-PID facilitates sharing, as well as interoperability.
This issue is much larger than merely an INF file only on Widows for a single protocol. For another example, only yesteday, I exchanged emails with someone wanting to use Teensy with a library on Andriod, which supported CDC-ACM and FTDI’s protocols. Sadly, the author identified compatible devices using a short list of VID-PID pairs. This guy was able to add another one to the list. But the point is depending on VID-PID pairs, rather than looking for information to identify the protocol, only serves to support a universe of non-interoperability and a world where obtaining a unique VID is of critical importance.
It’s unfortunate that most efforts to date have focused on attempting to change the USB-IF. That’s likely to be a fruitless effort. The USB-IF has no motivation to change, and every reason to reject proposals which threaten their source of sustaining revenue. While I don’t like the idea of paying $2k, I do really like the concept that a (at least in principle) vendor neutral organization is the steward of such an important technology standard.
Unlike the USB-IF, it *is* possible to influence open source and open hardware projects, especially from widely broadcast events like this open hardware summit. This talk, unfortunately, was an opportunity missed to advocate open hardware projects moving away from the conventional thinking about VID-PID pairs.
In the long term, the open source / open hardware community can migrate away from VID-PID dependence. When the VID number doesn’t matter, obtaining a unique VID goes from “problem” to “non-issue”.
It’s also quite possible, if eventually the entire open hardware world moves in this direction and creates lots of valuable, highly interoperatable devices and host implementations that become widely adopted, eventually the USB-IF may have a different perspective when dealing with the open source/hardware community. But today, nearly everyone follows the conventional VID-PID approach, which is terrible for interoperability, terrible at facilitating sharing, and when it comes to the USB-IF, only serves to reinforce their position to command $2k fees.
Ian, if you’re reading this, I hope you’ll consider it for future talks.
So you’re basically proposing to reinvent the USB device class mechanism in order to run a USB stack on top of the operating system’s USB stack, complete with dynamically loadable pluggable drivers? Good luck, let us know when you’ve got something working.
Every operating system already does a VID:PID independent system for handling MSC and HID devices you don’t need a new driver for every keyboard you plug in, it just sees the keyboard HID class and loads that driver. Linux has a rather independent CDC-ACM handler as well.
But the problem isn’t drivers, it’s obtaining a valid VID:PID pair. And whatever Rube Goldberg machinations are being proposed here do nothing to help the fact that the USB-IF will start sending nastygrams if you try to sidestep their license terms.
Exactly.
But there’s no need to be limited to only the subset of what Microsoft and Apple have provided (which is basically only HID and Mass Storage). Windows 7, after installing my INF for CDC-ACM also becomes as nice as Mac and Linux. All other CDC-ACM devices “just work”, as they do on Mac and Linux.
It is indeed possible, and not very difficult, to match against protocols instead of unique vid-pid identifiers. Probably the biggest obstacle is not technical, but the human factor where people set in their ways see this as a “Rube Goldberg machination”. The USB standard already provides these capabilities.
The primary problem is not technical, but cultural. The open source/hardware community is largely implementing USB in the traditional manner appropriate for proprietary products, which is the least suitable for sharing.
Rearding USB-IF nastygrams, today all open source USB code runs afoul of USB-IFs rules when used by anyone other than whom the VID number in that code is officially registered. Adapting device-side code to publish protocol information, and host-side code to match against protocols intead of unique identifiers does nothing to change that situation.
But moving in that direction on a large scale of most open projects will greatly lessen the pressure for non-profit or “kitchen table startup” uses to obtain a unique identifier. It’s also simply the right way to facilitate sharing. It’s unfortunate this point isn’t obvious.
I’d really like to understand what you’re proposing as. I follow this:
Windows 7, after installing my INF for CDC-ACM also becomes as nice as Mac and Linux. All other CDC-ACM devices “just work”, as they do on Mac and Linux.
Do you just mean that the INFs need to be properly constructed to pick up the device without the VID/PID match? That seems like no biggie, but isn’t some value in the USB ID field of the device descriptor required even if you’re matching device class from the driver instead of VID/PID?
Does the Teensy have a unique VID/PID pair?
I agree with you that USB CDC-ACM isn’t antiquated and use it in many projects. It just works, and it’s so easy to work with.
I’m not familiar with the Teensy insides, so I don’t fully understand the method you’re using. My understanding is that all USB devices need a VID/PID pair. That is a field in the device descriptor that must be filled in. I’ve never written any software that depends on the VID/PID pair to detect the device, so I’m not following how that applies to needing a unique ID. With CDC-ACM my programs only care what COM/serial port the device enumerated as. I’m by no means an expert on every aspect of USB though, I just use it to get stuff done in many projects.
I do object slightly to the assertion that my talk was a missed opportunity to tout your personal position :\ I’m glad it spurred discussion and I’m really happy to learn more about what you’re doing with Teensy though.
Paul, your work with Teensy 1/2/3 and Arduino libraries gives you a lot of USB experience, which I do not have. I have found is that (for example) my Windows boxes will often mistake some kinds of USB-connected GPS devices (eg. http://www.sureelectronics.net/goods.php?id=99 ), as a standard serial mouse, which of course causes various problems. I’ve seen others report this problem as well. Is it just a misconfiguration on a few Windows boxes, or some bad GPS designs, or a more general problem with generic USB classes?