Categories

Search Github projects by component, find design references

Posted on Thursday, November 12th, 2015 in components by DP

search-github

 

Search Github hardware by component to find examples and references for your next project. Big schematic and PCB previews mean quick and easy browsing without opening a CAD program.

TomKeddie first mentioned this idea at Hacker Camp Shenzhen, and later in the forum and on WeChat. Tom generously shared his scraping/search method. Eagle 6+ files are XML, so we can find them on Github by searching for the “eagle SYSTEM” tag in files with “extension:sch”. That gives more than the maximum 100 pages of results, so we filter by file size and increment size 500 bytes at a time “size:1001…1500”. We use the normal user search interface, parse the HTML results, and grab all urls ending with .sch. While Github has an API, that API doesn’t give access to search code search without specifying a repository by name (probably so people don’t do what we did…).

There are a number of limitations. Only Eagle projects are indexed, a search for KiCad files inspired little interest in expanding. Before Eagle 6.0 files were not XML and will not be included. Github search only indexes code files up to ~390K, larger files will not be included. If Github improves access via the API we will expand the index.

Scraping all of the files over terrible Chinese internet took about a week. Searches were performed at minimum 32 seconds apart. Some files were incomplete (lacked closing tag). Some files have extra ‘>>’, usually a merge error tag. Some files were just corrupt. These all caused Eagle to hang until the window was manually closed. Working with Eagle from the command line has been most unpleasant.

github-pcb

 

Via the forum.

 

 

This entry was posted on Thursday, November 12th, 2015 at 11:01 am and is filed under components. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

One Response to “Search Github projects by component, find design references”

  1. J. Peterson says:

    This is very cool. It’s probably worth noting that Github being what it is, you may be looking at somebody’s half-baked work in progress. But even that can still beat starting from scratch or answer questions the spec sheets don’t.

Leave a Reply

Notify me of followup comments via e-mail. You can also subscribe without commenting.

Recent Comments

  • KH: So Romanians use old ICs too, a bit reminiscent of Dilshan's projects. I'm vaguely disappointed by the liberal use of float data and float calculations....
  • Max: This is of course no "0-30V" supply - the single-supply standard LM723 schematic can't go all the way down to zero, which is why I...
  • KH: A power-off current draw of 4mA for the Sony A6000 is a defect, product recall territory. It hits the camera battery with unacceptable wear and...
  • Olav Storstrøm: Please add 9-pin D-sub RJ45 and RJ12 connectors! Thanks!
  • KH: Hydroponics is like home automation. Every year you have folks who start projects like these and think it will take off and become a big...