tag:blogger.com,1999:blog-2570785179139408652024-03-13T21:41:19.256-03:00/dev/stdoutAnonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-257078517913940865.post-86173677421631757242016-01-11T12:29:00.000-03:002016-01-11T12:30:03.969-03:00An Overview of the Upcoming libModSecurity<span style="background-color: white; font-family: Georgia; font-size: 14px;"><span style="color: red;">Notice:</span> This blog post was originally posted on<a href="https://www.trustwave.com/Resources/SpiderLabs-Blog/ModSecurity-Python-Bindings--Parsing-ModSecurity-rules-from-Python/?page=1&year=0&month=0"> SpiderLabs Blog</a>.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">libModSecurity is a major rewrite of ModSecurity. It preserves the rich syntax and feature set of ModSecurity while delivering improved performance, stability, and a new experience in easy integration on different.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<br />
<h2 style="background-color: white; font-family: Georgia;">
<strong>libModSecurity - Motivations</strong></h2>
<div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
While ModSecurity version 2.9.0 is available on different platforms (IIS, NGINX, etc…), It really favors an Apache Deployment. ModSecurity standalone is part of ModSecurity project and is basically a wrapper that packs requests from different formats into an Apache format, to later be processed by ModSecurity in the same fashion that it works on an Apache web server. That was certainly the fastest way to have ModSecurity running on different platforms but at the cost of performance and high amount of dependencies. Leading, for instance, to situations where NGINX users have to install Apache dependency in order to have the NGINX ModSecurity module working; see metabug <a href="https://github.com/SpiderLabs/ModSecurity/issues/661" style="color: #7e57c2; position: relative; z-index: 0;" target="_blank">ModSecurity/#661</a> for further information. The growth of the project in terms of integration with scripting languages or adoption in other platforms such as IDSs becomes very difficult because of those limitations.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
The addition of new operators or general features is also limited because ModSecurity 2.9 (Sec Language) uses the Apache configuration parser to be loaded. In other words, the language format and conditional syntax could not be extended because it was dependent on a 3rd party software, that may not have our features in mind when they make changes. Even worse, sometimes bugs that heavily affect us are not on the priority list of the Apache (<a href="https://bz.apache.org/bugzilla/show_bug.cgi?id=55910)" style="color: #7e57c2; position: relative; z-index: 0;" target="_blank">https://bz.apache.org/<wbr></wbr>bugzilla/show_bug.cgi?id=<wbr></wbr>55910)</a>.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
</div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
<strong>LibModSecurity – Our goals</strong></h2>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">In order to circumvent those limitations, a year ago we decided to develop a new version of ModSecurity. A version for which the primary goal was not to introduce new features, but to add the possibility of an easy expansion and integration.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Due to the limitations of “ModSecurity standalone” and ModSecurity v2.9 architecture, we decided to move forward to implement something from scratch. By providing a new architecture while simultaneously supporting all the features of 2.9 we were able to remove many of the limitations that have burdened the project for the past few years.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
In this blog post I plan to go over some of the details on why it was important to change the architecture and how some of those changes will lead to positive outcomes. Enjoy...</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
</div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
More about ModSecurity standalone architecture</h2>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">As explained previously, the ModSecurity 2.x version has a high dependency on 3rd party projects including Apache. As it uses Apache internals directly, it was also using the libapr (Apache Portable Runtime). To highlight this point, we note that almost all memory allocation inside ModSecurity was allocated using libapr memory pools. This is, of course, not a problem, unless you want to remove APR.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">To understand a little bit about the level of dependency of ModSecurity version 2.9 on 3rd party modules have a look on the </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 1</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisQihvWLg08aLpXA135DfmoZq-IkgzmDETIuqDlU71xl2Sf5cyCjspf19syep9dzOMrMbwbrh7Lg_PTOl3pWn8yynLy1R8STpyqj4yf_rQ67NlGMY0dQyvyGY29VoPCxZqNi28cEf1HGk/s1600/6a0133f264aa62970b01b7c7fe12d5970b.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisQihvWLg08aLpXA135DfmoZq-IkgzmDETIuqDlU71xl2Sf5cyCjspf19syep9dzOMrMbwbrh7Lg_PTOl3pWn8yynLy1R8STpyqj4yf_rQ67NlGMY0dQyvyGY29VoPCxZqNi28cEf1HGk/s400/6a0133f264aa62970b01b7c7fe12d5970b.png" width="400" /></a></div>
<div>
<strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 1.</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;"> ModSecurity dependencies. (a) nginx extension. (b) IIS extension. Notice that both utilize the ModSecurity Standalone module.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">At the right of </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 1 </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">we have several dependencies that are bound to the operators or request body parsers (e.g. libxml), some of those are mandatory on ModSecurity version 2.9.x and the idea is that those will become optional in ModSecurity version 3. The availability of a given feature will depend on the existence of the dependency. But, that is the subject for another blog post :)</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">On the left side of the </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 1 </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">are listed the dependencies that are mandatory for the ModSecurity v2.9.x core execution, without these core features of ModSecurity v2.9 cannot function. Removing the APR without removing other Apache dependencies was not be possible, as there are internal calls to the Apache API which demand data in the APR structure (APR memory pool). Additionally, removing just the Apache dependency also did not make sense, as it would need to be replaced by something Apache-like in any event, inheriting all those limitations. So the natural step was to move to a new architecture free of both Apache and APR. </span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
libModSecurity</h2>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The first thing that comes to mind when discussing “refactoring” of the ModSecurity core, is the possibility to have a segmentation of what is the “core” and what code is required to interact with a given web server, or as we call it a “connector”. Looking at the issues on GitHub it is difficult to tell which bug reports belong to each part of the code.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">This monolithic design also puts pressure on us, its maintainers, when developing, testing, and packaging releases. This is because whenever we release in the current version, 2.x, everything needs to be released together (All platforms, even if there is no benefit to a given platform). Splitting the core from the “connectors” seems to be the right choice. Splitting the project give us numerous advantages, not only from the project planning perspective, but also the fact that the library can be easily ported, , and manipulated in this more modular architecture.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">By splitting the project between “connectors” and “core”, the core naturally become a </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">library</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">, and the connectors become consumers of the core library. This way, ModSecurity core becomes completely independent of the underlying web server.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Another goal of the project was to reuse as much code as possible; this was done for two main reasons: (1) the code was tested and proven to work. (2) we didn’t need to implement everything from the scratch. So we first implemented a very minimalistic LALR parser (SecLanguage) to read the rules and transform them into C++ objects in memory. That limited language was slowly expanded as we added more operators, transformations and variables.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">During the development of the core, other important utilities were also created. Two of those utilities deserve special attention, namely, the </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">regression</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;"> and </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">unit test</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;"> utilities. Both have a very important role in the development of ModSecurity version 3 and probably they will be even more important for the maturity of the project. In the future there will be a specific blog post to cover the regression and unit tests inside ModSecurity version 3.</span></div>
<div>
<br /></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">One of the challenges on ModSecurity version 3 was to rewrite the SecRules, but to return exactly the same results of ModSecurity v2.9.x, thereby avoiding a compatibility break. This meant that we also had to reproduce the corner cases and unexpected behaviors of v2.9.x. This was achieved, so far, by the utilization of the regressions tests, were the requests are mimicked into JSON and the expected results are established from analysis of ModSecurity 2.9.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Going beyond the test, we needed to see ModSecurity version 3 working in practice as part of a web server. We choose the NGINX web server to be the first to fully implemented ModSecurity version 3. For that we started a different GitHub project, called ModSecurity-nginx (</span><a href="https://github.com/SpiderLabs/ModSecurity-nginx" style="background-color: white; color: #7e57c2; font-family: Georgia; font-size: 14px; position: relative; z-index: 0;" target="_blank">https://github.com/<wbr></wbr>SpiderLabs/ModSecurity-nginx</a><span style="background-color: white; font-family: Georgia; font-size: 14px;">).</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<strong style="font-family: Georgia; font-size: 14px;">Figure 2</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;"> contains the dependencies of “ModSecurity NGINX connector”.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiEBU3j2omnDF03BY7ca2WaxhlZ4HDjYwXXCgqykgKWvTfraXS7QY6k7mAkRJZaUnnu7Ht2MS0L_etX0z998kYXHuhjLzhcRUTxV2yRwMscC4iZBZghFRHQESOhClv-C1_lb7ElT8wI9s/s1600/6a0133f264aa62970b01b7c7fe1307970b.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiEBU3j2omnDF03BY7ca2WaxhlZ4HDjYwXXCgqykgKWvTfraXS7QY6k7mAkRJZaUnnu7Ht2MS0L_etX0z998kYXHuhjLzhcRUTxV2yRwMscC4iZBZghFRHQESOhClv-C1_lb7ElT8wI9s/s400/6a0133f264aa62970b01b7c7fe1307970b.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<strong style="background-color: white; font-family: Georgia; font-size: 14px;">F</strong><strong style="background-color: white; font-family: Georgia; font-size: 14px;">igure 2.</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;"> Dependencies of ModSecurity NGINX connector for libmodsecurity.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div style="background-color: white; font-family: Georgia;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The ModSecurity NGINX connector, together with libModSecurity was presented at </span><a href="https://www.nginx.com/nginxconf/schedule/#day2s30" style="background-color: white; color: #7e57c2; font-family: Georgia; font-size: 14px; position: relative; z-index: 0;" target="_blank">NGINX.conf 2015</a><span style="background-color: white; font-family: Georgia; font-size: 14px;">. At the conference we discussed various aspects of the ModSecurity connector and we obtained valuable feedback from the community to shape development of the connector.</span></div>
<div style="background-color: white; font-family: Georgia;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div style="background-color: white; font-family: Georgia;">
</div>
<h2>
<strong>Current status?</strong></h2>
<br />
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Currently we have the major features of libModSecurity implemented and ready to use. Although we don’t have support for collections yet, it is possible to load the OWASP core rule set version 3.0.0-dev.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">So far, we are 374 commits ahead of master. Supporting almost all ModSecurity 2.x features. </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 3. </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">Contains a punch card with the commits to the project so far.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2vmoUg8DRmaPAB_y9loFQngweK-tb9zZBhGvcERgTlMkgH14h_N06BrUwH4HvGHSRMUk1a_azJodu9QIFX8CsoXE-JyMyZE2Gr83qX35paNQ7oGftk5s9-FxBn8siA9xZROvETFVlM4Q/s1600/6a0133f264aa62970b01bb08a2a0b3970d.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="148" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2vmoUg8DRmaPAB_y9loFQngweK-tb9zZBhGvcERgTlMkgH14h_N06BrUwH4HvGHSRMUk1a_azJodu9QIFX8CsoXE-JyMyZE2Gr83qX35paNQ7oGftk5s9-FxBn8siA9xZROvETFVlM4Q/s400/6a0133f264aa62970b01bb08a2a0b3970d.png" width="400" /></a></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"> </span><strong style="font-family: Georgia; font-size: 14px;">Figure 3. </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">Commits punch card.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
Testability</h2>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">While starting with versions 3, one of the main concerns was the project quality and that is why most of the features contain regression tests and/or unit tests. These tests can be executed on the developer’s machine but they are also executed on our BuildBots.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">ModSecurity version 2.9.x already has its own regression and unit test utilities, however, they are very slow. A test with these utilities will take around 1 hour, has a dependency on Perl scripts, is Unix only, and is dependent on the web server.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">With version 3 we fork the tests used in 2.9 into a separate GitHub project, and we added it inside ModSecurity by the utilizing a Git subtree. Having the test cases migrated into a separate project gives us the flexibility to run different tests with different versions of ModSecurity.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">We also created another subset of tests that is specific to the connector. This was done so we can segregate potential problems between the core and any connectors. In the specific case of NGINX, the regression tests were built as an extension of the NGINX server regression tests.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">All these tests are executed for each commit that is performed on GitHub. This gives us the capability to promptly identify and fix any problem during the development of ModSecurity v3. It also gives us the ability to mimic problems without needing an entire web server configuration.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjidxidHodfAhyphenhyphen_cTuVjXsjxH-pGrlGGwdMLLw2M2UJ0xY0YZVO3bwzsTvdln-iMoEKRg0RiBjaRMGSG1d2AhK9LwRiGIJz47fU2uAGy1Kn8Xz-1fgYQbAppAXREK58bf2PqGeXgIvMyGs/s1600/6a0133f264aa62970b01bb08a2a04d970d.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="103" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjidxidHodfAhyphenhyphen_cTuVjXsjxH-pGrlGGwdMLLw2M2UJ0xY0YZVO3bwzsTvdln-iMoEKRg0RiBjaRMGSG1d2AhK9LwRiGIJz47fU2uAGy1Kn8Xz-1fgYQbAppAXREK58bf2PqGeXgIvMyGs/s400/6a0133f264aa62970b01bb08a2a04d970d.png" width="400" /></a></div>
<div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<strong>Figure 4. </strong>Buildbot in the shape that we want to see, all green.</div>
</div>
<div>
<br /></div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
Performance</h2>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Performance is something that we are constantly measuring during the development of version 3. Instead of using the DebugLogs, with performance time stamps, we created SystemTap scripts that allow for real time instrumentations. SystemTap also allows the creation of flame charts. As demonstrated on </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 5.</strong></div>
<div>
<strong style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></strong></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUcE9J1H7QKQb-sp7OGTNjcXM4UJY9uYmy8JkNC1s3cleZDob3Kmj_efm4W0tkJ8gfu_I6lnSDALlxEBof0Ube45duXugNLgjacDp2ktYlX7apZKkDCXg7drJq4HtsxQ153vhh3SxT84g/s1600/6a0133f264aa62970b01b7c7fe132b970b.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUcE9J1H7QKQb-sp7OGTNjcXM4UJY9uYmy8JkNC1s3cleZDob3Kmj_efm4W0tkJ8gfu_I6lnSDALlxEBof0Ube45duXugNLgjacDp2ktYlX7apZKkDCXg7drJq4HtsxQ153vhh3SxT84g/s400/6a0133f264aa62970b01b7c7fe132b970b.png" width="400" /></a></div>
<div>
<strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 5. </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">ModSecurity version 3 execution time using OWASP CRS version 3.0.0-dev.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 5 </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">illustrates the mean times of each rule, grouped into different phases. Notice that the entire execution of the OWASP CRS took 483 microseconds. That specific subject deserves a creation of a specific blog post, together with the possibility of rules optimization.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
What is next?</h2>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">As mentioned, the libModSecurity still isn’t feature complete when compared to ModSecurity version 2.9.x. Your help is more than welcome to support that initiative. Currently, we have missing Connectors, Operators, Transformations, and Variables. How do you want to help?</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<h3 style="background-color: white; font-family: Georgia;">
Connectors</h3>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">As mentioned before, currently we only have connector for nginx. In other words, Apache users will not have any benefit from ModSecurity v3, at least, not yet.</span></div>
<div>
<br /></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The development of the connectors for IIS and Apache will start as soon as we have a version of ModSecurity v3 released, unless someone from the community starts to develop it :)</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<h3 style="background-color: white; font-family: Georgia;">
Operators, Transformations and Variables</h3>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">A question that we frequently receive is how to start coding for ModSecurity. Well, this is a good opportunity. The missing features in ModSecurity versions 3 are not very difficult to, plus, many of them already contain a description of what needs to be done. Some even have the file already, and just need to be filled. For a complete list of missing features with descriptions, check here:</span></div>
<div>
<br />
<ul>
<li><a href="https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20transformation">Missing transformations</a></li>
<li><a href="https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20variables">Missing variables</a></li>
<li><a href="https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20operators">Missing operators</a></li>
<li><a href="https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20features">Missing features</a></li>
<li><a href="https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20documentation">Missing documentation</a></li>
</ul>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Although all the operators needed by the ModSecurity SLR commercial rules are already supported inside ModSecurity version 3, we just encourage the usage of ModSecurity version 3 to advanced users. ModSecurity version 3 was not released yet, thus, not considered stable. Once it is released it will be 100% compatible with our SLR commercial rules.</span></div>
<div>
<br /></div>
<div>
<h3 style="background-color: white; font-family: Georgia;">
Testing!</h3>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">At this stage of the development testing is very important. We count on the community to provide feedback and report any issue on ModSecurity version 3. We are aiming to have a release candidate soon, ideally with only a small number of issues.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com1tag:blogger.com,1999:blog-257078517913940865.post-26702332430641145082016-01-11T12:26:00.000-03:002016-01-11T12:30:08.613-03:00ModSecurity Python Bindings: Parsing ModSecurity rules from Python<span style="background-color: white; font-family: Georgia; font-size: 14px;"><span style="color: red;">Notice:</span> This blog post was originally posted on</span><a href="https://www.trustwave.com/Resources/SpiderLabs-Blog/ModSecurity-Python-Bindings--Parsing-ModSecurity-rules-from-Python/?page=1&year=0&month=0" style="font-family: Georgia; font-size: 14px;"> SpiderLabs Blog</a><span style="background-color: white; font-family: Georgia; font-size: 14px;">.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">One of the good things about the next generation of ModSecurity, libModSecurity (AKA ModSecurity version 3), is the fact that it portable to almost any platform. This extensibility makes the use of bindings for other languages, beyond C/C++, very useful. For those that are unfamiliar with the concept of a ‘binding’, the term describes an interface between two different languages. In the case of ModSecurity this binding will provide an interface that allows you to use libModSecurity functionally inside your scripting language of choice, with negligible loss of performance.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The libModSecurity Python bindings may serve many purposes; for instance, it could be leveraged to rapidly (and easily) create a custom web interface for ModSecurity. However, we are not limited to simply displaying information, the new bindings give you nearly all the capabilities that you would have interacting with libModSecurity from native C, allowing you to, for example, display ModSecurity rules in any fashion that you desire, with almost zero effort. A simple usage is demonstrated on the </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 1</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc2x30yOPsKocO7DIaQe752lNQ79FIJhxJNN2spNuY502yfEgnvcp8qhC8moV_63Zn7RHML_sc8cMZFCTykTMFFgxzfjpThKL_IkenEmaNhpD1N60D9HcOyDamZH_JB2Iu7Sv6ZfxdChE/s1600/6a0133f264aa62970b01bb08a2a10a970d.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc2x30yOPsKocO7DIaQe752lNQ79FIJhxJNN2spNuY502yfEgnvcp8qhC8moV_63Zn7RHML_sc8cMZFCTykTMFFgxzfjpThKL_IkenEmaNhpD1N60D9HcOyDamZH_JB2Iu7Sv6ZfxdChE/s400/6a0133f264aa62970b01bb08a2a10a970d.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 1.</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;"> ModSecurity being used inside a Python shell.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Bindings for languages others than Python will be also created. In fact, the utility that was used in the creation of the Python bindings, SWIG, can also generates interfaces to other languages such as: Ruby. You are welcome to volunteer to expand your favorite language to add ModSecurity capabilities.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;">In this blog post I will dive into the creation of a simple console utility to load and list a set of rules in a elegant way. For this I will start with the compilation of the ModSecurity-Python-bindings.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Notice: The ModSecurity Python bindings depend on the libModSecurity which is publicly available on ModSecurity GitHub, but it is not considered stable release at this point yet.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<h2 style="background-color: white; font-family: Georgia;">
Installing ModSecurity Python Bindings</h2>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Before starting the installation of the ModSecurity Python bindings, make sure you have libModSecurity installed and operational on your machine. For further information about libModSecurity and how to install it, please see the README at ModSecurity’s GitHub repository, here: </span><a href="https://github.com/SpiderLabs/ModSecurity/tree/libmodsecurity" style="background-color: white; color: #7e57c2; font-family: Georgia; font-size: 14px; position: relative; z-index: 0;" target="_blank">https://github.com/<wbr></wbr>SpiderLabs/ModSecurity/tree/<wbr></wbr>libmodsecurity</a><span style="background-color: white; font-family: Georgia; font-size: 14px;"> . Of course, you will also want to make sure you have Python installed and all the developer utilities you may need on your Linux .</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Once libModSecurity is installed, it is time to proceed with the installation of the ModSecurity Python bindings. Simply download the code from our GitHub repository, and proceed with the compilation, as demonstrated below:</span></div>
<div>
<pre style="background-color: white; font-size: 14px;"> $ git clone <a href="http://www.github.com/SpiderLabls/ModSecurity-Python-bindings" style="color: #7e57c2; position: relative; z-index: 0;" target="_blank">http://www.github.com/<wbr></wbr>SpiderLabls/ModSecurity-<wbr></wbr>Python-bindings</a>
$ cd ModSecurity-Python-bindings
$ make
$ sudo make install</pre>
<pre style="background-color: white; font-size: 14px;"></pre>
<pre style="background-color: white; font-size: 14px;"></pre>
</div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Once this process is finished, it is a good idea to test your new installation. You can do this by running the test script provided. This can be accomplished using the following command:</span></div>
<div>
<pre style="background-color: white; font-size: 14px;"> $ ./test/t.py</pre>
<pre style="background-color: white; font-size: 14px;"></pre>
<pre style="background-color: white; font-size: 14px;"><span style="font-family: Georgia; white-space: normal;">If everything is alright, you should not get any error messages from Python.</span></pre>
<pre style="background-color: white; font-size: 14px;"><span style="font-family: Georgia; white-space: normal;">
</span></pre>
<pre style="background-color: white; font-size: 14px;"><span style="font-family: Georgia; white-space: normal;">
</span></pre>
</div>
<div>
<h2 style="background-color: white; font-family: Georgia;">
The hello world script</h2>
</div>
<div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
With the ModSecurity Python bindings installed and tested it is time to create your first ‘hello world’ application using the ModSecurity library. Let’s start with something very simply like checking the ModSecurity version that we are playing with.</div>
</div>
<div>
<br /></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Inside libModSecurity we expose the whoAmI() method which describes which version of ModSecurity you are bound with. Further information about this method, can be found here: </span><a href="https://github.com/SpiderLabs/ModSecurity/blob/libmodsecurity/src/modsecurity.cc#L67-L78" style="background-color: white; color: #7e57c2; font-family: Georgia; font-size: 14px; position: relative; z-index: 0;" target="_blank">https://github.com/<wbr></wbr>SpiderLabs/ModSecurity/blob/<wbr></wbr>libmodsecurity/src/<wbr></wbr>modsecurity.cc#L67-L78</a></div>
<div>
<br /></div>
<div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
Let’s use the <em>whoAmI</em>() method, to print the libModSecurity version. We demonstrate proper usage in <strong>Script 1 </strong>below.</div>
</div>
<div>
<br /></div>
<div>
<pre class="prettyprint linenums">#!/usr/bin/python
from modsecurity import *
modsec = ModSecurity()
s = “Hello World, I am: “ + str(modsec.whoAmI())
print s
</pre>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<strong>Script 1.</strong> Hello world using libModSecurity.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
As output, you should be able to see something similar to the <strong>Figure 2.</strong></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<strong><br /></strong></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9xJtSUpysU7pyi7dAoj14tZF32h2Bn9-dH3urQiJvx7xXUZeJq9LcCrCF7FVY0k8iM6Ydl3CvmmovQErHFPRUv4Z_uiifEPhx9As20qj2hDD5kTnVdYEi-Pkid8381lMxAZ1-ErbDjVY/s1600/6a0133f264aa62970b01bb08a2a1a3970d.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="52" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9xJtSUpysU7pyi7dAoj14tZF32h2Bn9-dH3urQiJvx7xXUZeJq9LcCrCF7FVY0k8iM6Ydl3CvmmovQErHFPRUv4Z_uiifEPhx9As20qj2hDD5kTnVdYEi-Pkid8381lMxAZ1-ErbDjVY/s400/6a0133f264aa62970b01bb08a2a1a3970d.png" width="400" /></a></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<strong>Figure 2. </strong>The output of the hello world script.</div>
<h2 style="background-color: white; font-family: Georgia;">
</h2>
<h2 style="background-color: white; font-family: Georgia;">
Loading the rules</h2>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
The next step is to add a few more pieces to our script allowing it to actually load a given set of rules. Make sure you have a workable set of ModSecurity rules before you start to code. A good set for this example might be the OWASP CRS 3.0 ruleset, available here: <a href="https://github.com/SpiderLabs/owasp-modsecurity-crs/tree/v3.0.0-dev" style="color: #7e57c2; position: relative; z-index: 0;" target="_blank">https://github.com/<wbr></wbr>SpiderLabs/owasp-modsecurity-<wbr></wbr>crs/tree/v3.0.0-dev</a></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
Notice: Make sure you download the version 3.0.0-dev. Our parser is not 100% compatible with ModSecurity v2.9.x yet. It may not work with other versions of CRS.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
Using our first example, start by commenting out the “Hello World...” string and the associated print statement. These two pieces won’t be necessary for this step. Your script should look similar to example provided in <strong>Script 2</strong>.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<pre class="prettyprint linenums">#!/usr/bin/python
from modsecurity import *
modsec = ModSecurity()
#s = “Hello World, I am: “ + str(modsec.whoAmI())
#print s
</pre>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<strong>Script 2. </strong>Hello World script with the unnecessary strings commented.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
The rules in ModSecurity are loaded through a Rules object. While the Rules object may be merged with other objects of the same type, in this script let’s keep it simple. For this example we just need to load a set of rules from a file and print them to the console. Again, a very simple utilization for libModSecurity.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
We can load the rules, using the <em>loadFromUri() </em>method, which takes one argument as follows: “loadFromUri('/path/to/the/<wbr></wbr>rules/file.txt')”. The <em>loadFromUri()</em> method allow us to load a set of rules into memory. Notice that this method will return “-1” if there are any problems while loading the rules. If this occurs you can call the <em>getParserError()</em> method to get more information about any possible errors. The <em>loadFromUri()</em> method must be called with the path to a rules file as a target, as demonstrated in <strong>Script 3</strong>.</div>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<br /></div>
<pre class="prettyprint linenums">#!/usr/bin/python
from modsecurity import *
modsec = ModSecurity()
#s = “Hello World, I am: “ + str(modsec.whoAmI())
#print s
rules = Rules.loadFromFile(“/path/to/<wbr></wbr>your/v3.0.0-dev/rules/REQUEST-<wbr></wbr>10-IP-REPUTATION.conf”)
</pre>
<div style="background-color: white; font-family: Georgia; font-size: 14px;">
<strong>Script 3. </strong>Loading the rules from a given file.</div>
</div>
<div>
<br /></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The Rules object has provides the </span><em style="background-color: white; font-family: Georgia; font-size: 14px;">getRulesForPhase()</em><span style="background-color: white; font-family: Georgia; font-size: 14px;"> method which is called as follows: “rules.getRulesForPhase(phase_</span><wbr style="background-color: white; font-family: Georgia; font-size: 14px;"></wbr><span style="background-color: white; font-family: Georgia; font-size: 14px;">number)”. Calling this method will return a vector of Rule objects. The Rule object contains all the properties associated with a given rule. To list all the rules from the target file that will load during a given phase you may simply iterate over the returned value from </span><em style="background-color: white; font-family: Georgia; font-size: 14px;">getRulesForPhase</em><span style="background-color: white; font-family: Georgia; font-size: 14px;">(). This is demonstrated in </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Script 4</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<pre class="prettyprint linenums">#!/usr/bin/python
from modsecurity import *
modsec = ModSecurity()
#s = “Hello World, I am: “ + str(modsec.whoAmI())
#print s
rules = Rules()
r = rules.loadFromUri(“/path/to/<wbr></wbr>your/v3.0.0-dev/rules/REQUEST-<wbr></wbr>10-IP-REPUTATION.conf”)
if r == -1:
print rules.getParserError()
sys.exit()
i = 0
while i < modsec.NUMBER_OF_PHASES:
r = rules.getRulesForPhase(i)
print "-- Phase " + str(i)
for x in r:
if x.rule_id == 0:
continue
print " Rule Id: " + str(x.rule_id)
print " From: " + str(x.m_fileName) + " at " + str(x.m_lineNumber)
i = i + 1</pre>
<pre style="background-color: white; font-size: 14px;"><strong style="font-family: Georgia; white-space: normal;">Script 4. </strong><span style="font-family: Georgia; white-space: normal;">Print all rules ID from a ModSecurity configuration file. </span></pre>
<pre style="background-color: white; font-size: 14px;"><span style="font-family: Georgia; white-space: normal;">
</span></pre>
<pre style="background-color: white; font-size: 14px;"><span style="font-family: Georgia; white-space: normal;">The output of </span><strong style="font-family: Georgia; white-space: normal;">Script 4</strong><span style="font-family: Georgia; white-space: normal;"> should appear similar to the output illustrated below in</span><strong style="font-family: Georgia; white-space: normal;"> Figure 3</strong><span style="font-family: Georgia; white-space: normal;">.</span></pre>
<pre style="background-color: white; font-size: 14px;"><span style="font-family: Georgia; white-space: normal;">
</span></pre>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzFBfONe3gaNFQpDUfEm30-Dlxu0iYEG63GAedE-dxLspOvIx_H8Oy1W2xpv0h_nszENhOzG7ZpmXzRDfXOXsccSKb1dTBepw_Rc8-FJ-8jPURpYM5nPGaDKnEQ8GJALHurDyzCsl_yX8/s1600/6a0133f264aa62970b01b7c7fe1697970b.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzFBfONe3gaNFQpDUfEm30-Dlxu0iYEG63GAedE-dxLspOvIx_H8Oy1W2xpv0h_nszENhOzG7ZpmXzRDfXOXsccSKb1dTBepw_Rc8-FJ-8jPURpYM5nPGaDKnEQ8GJALHurDyzCsl_yX8/s400/6a0133f264aa62970b01b7c7fe1697970b.png" width="400" /></a></div>
<strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 3. </strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">The output of the script which prints the rules ID in the console.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Of course this script can be extended to print this information in any manner desired. A slightly more refined example is illustrated in </span><strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 4</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">. Remember that as this is Python it is easy to display this information as part of GUI or web application.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib0dxhcAs7HN4UeTwZix-UhAU5JwMeEy7NlgrIJnUGe28IaZg7m9TJ2KHC9At9Z8M2XT1mROphDpX2k9i9bB_-zRnJVOEzZ8BBHejmuWmBqh07SiYVXPbBfxubcssx7Tje9VJEAfyHMFk/s1600/unnamed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib0dxhcAs7HN4UeTwZix-UhAU5JwMeEy7NlgrIJnUGe28IaZg7m9TJ2KHC9At9Z8M2XT1mROphDpX2k9i9bB_-zRnJVOEzZ8BBHejmuWmBqh07SiYVXPbBfxubcssx7Tje9VJEAfyHMFk/s400/unnamed.png" width="400" /></a></div>
<strong style="background-color: white; font-family: Georgia; font-size: 14px;">Figure 4</strong><span style="background-color: white; font-family: Georgia; font-size: 14px;">. ModSecurtity rules information extracted using the libModSecurity parser, inside a Python script.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span>
<br />
<h2 style="background-color: white; font-family: Georgia;">
Conclusions</h2>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The new ModSecurity Python bindings should make it easy and fast to utilize libModSecurity as demonstrated in the examples above. These Python bindings are just one of a host of new features we will be presenting in upcoming blog posts, so be on the lookout.</span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;"><br /></span></div>
<div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">The advantage of fast prototyping provided by the script language utilization, plus, the performance of a core in C++ opens a wide range of possibilities. Like the construction of a Rule editor in Python, and/or a Django web application to navigate inside the rules. In fact, why not implement those as open source projects?</span></div>
<div>
<br /></div>
<span style="background-color: white; font-family: Georgia; font-size: 14px;">Notice that those are just simple examples, but it is just as easy to use these bindings to actually process a transaction, but this is left as exercise for the reader, enjoy :)</span><br />
<br />Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-65873593392869419962013-01-29T07:19:00.000-03:002013-09-10T14:27:31.260-03:00Exploring GSM Vulnerabilities to Assess Mobile Users' Location<div>
<div style="text-align: justify;">
The privacy of about 5 billion GSM users worldwide is exposed. Their location is available to eavesdroppers while society keeps attention to vulnerabilities in software layers which may compromise the privacy of a very select group of users. The GSM networks depend on the mobile phones location to provide a good service. That dependency is intrinsic to the GSM protocol, where the phone must be near to an antenna in order to ensure a good radio signal quality between the mobile phone and the Base Transceiver Station. Unlike social networks or other Internet services where the users need to explicitly authorize the sharing of information about their location, the GSM protocol does it ubiquitously.</div>
</div>
<div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Using as excuse my master's degree, in the past years I've been fully entertained studying/researching about some GSM vulnerabilities that may expose the geolocation of users. During the work I've studied known vulnerabilities and I also made some patches on known GSM security tools which may be very handful to reproduce the experiments I've made to prove my point.</div>
<div style="text-align: justify;">
<br /></div>
</div>
<div>
<div style="text-align: justify;">
My work was divided into three different ways to assess the user geolocation, they are unrelated but information gather in one may be useful to others attacks. These three different attacks intend to collect the victim geolocation in the following accuracy levels: <strong>country</strong>, <strong>portion of the city</strong> and <strong>neighborhood</strong>. This may vary depending on the provider configuration. So, do not expect this to work on all cases.</div>
</div>
<div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To illustrate the attacks we use two different entities: the <em>attacker</em> and the <em>victim</em>. For the second and third attacks, besides knowing the mobile number of the <em>victim</em>, the attacker also needs to have two <a href="http://bb.osmocom.org/" target="_blank" title="OsmocomBB">OsmocomBB</a> compatible mobile phones and some specific <a href="http://bb.osmocom.org/" target="_blank" title="OsmocomBB">OsmocomBB</a> patches that are explained in this post. For the first attack, the attacker just need to have an account in an HLR provider and, of course, the mobile number of the <em>victim</em>.</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div>
<h3>
<strong>First attack, Country</strong></h3>
<div>
<strong><br /></strong></div>
<div style="text-align: justify;">
By knowing the telephone number of the victim it is possible to track down its country of origin by the phone prefix. It is also possible to know if the number still valid, working and if the user is in roaming or not. If so, which country the user is.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
All this information can be gather using an HRL query provider. Note, however, that just some providers are leaking such information. In some countries (like Brazil) the providers are not able to leak this information, due law enforcement. In the map bellow is possible to identify in which countries the information is available (in green) or not (in orange).</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgJclBPB9Ymr5E-GDzQ6UbefX-o9ZI5H06CsSeRDP_vMJmrd_23M-qegd1zF-B77g0ospq8KMBV-md_iTPtzBvxoKfUf_Uqg1EEaOe8dQbsd5vGumXEgpSzaA224YZa7tO_u_njav3g_s/s1600/mapUsers-1024x650.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgJclBPB9Ymr5E-GDzQ6UbefX-o9ZI5H06CsSeRDP_vMJmrd_23M-qegd1zF-B77g0ospq8KMBV-md_iTPtzBvxoKfUf_Uqg1EEaOe8dQbsd5vGumXEgpSzaA224YZa7tO_u_njav3g_s/s400/mapUsers-1024x650.png" width="400" /></a></div>
<br />
<div style="text-align: justify;">
Depending on your HLR provider, the information about the country comes already parsed, as illustrated in the example bellow.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNGvDYCD9ZQzLcV710k07TXk5bbxsVcxyiT3hXO4QXe-HLzJX887TnC5tYmK84LlDcusxSjRysj2DqswYni7M6j2avLRPbRkU7Ul1DjbWXyoTUxLUKmGI-FMhdBHr7gI42n-msg94nnQc/s1600/hlr.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNGvDYCD9ZQzLcV710k07TXk5bbxsVcxyiT3hXO4QXe-HLzJX887TnC5tYmK84LlDcusxSjRysj2DqswYni7M6j2avLRPbRkU7Ul1DjbWXyoTUxLUKmGI-FMhdBHr7gI42n-msg94nnQc/s400/hlr.png" width="400" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As <em>further work</em>, the MSC gateways could be used to map the geolocation of the users, however, a mapping of the region is needed to be done prior to the attack.</div>
<br /></div>
<br />
<h3>
<strong>Second attack, Portion of the city</strong></h3>
<div>
<strong><br /></strong></div>
<div>
<div style="text-align: justify;">
This attack relays on the fact that a mobile phone is always connected to a BTS, listening a broadcast channel which belongs to a virtual group called: LAC. Knowing that, the <em>attacker</em> can address some demand (<strong>SMS</strong> or <strong>call</strong>) to the <em>victim</em> phone number and wait to see the <em>victim</em> identification (<strong>IMSI</strong> or <strong>TMSI</strong>) being redirected by the network to an specific channel/time slot to receive the demanded content. Broadcast channels are always delivering messages to users, more than one per second which makes impossible to the attacker to identify the victim <strong>TMSI</strong> or <strong>IMSI</strong> in the first try, so subsequent attempts need to be made reducing a `possible-victim list`, narrow it to one if the victim is part of a LAC or zero if the victim is not part of it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Bellow there is a graphic of my hometown, Recife, with the LACs in different colors. It gives us an idea how big the LACs are and consequently the accuracy of the attack.</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9XxbYgHSu2kv-cIalLyC8DYW8LaqaUryD4qeS4CaQ6-yVihCK9J7Eel5VYnHHkRvylA6IvAl-faZtIOwgKpxFtp3uLuBrB_rU1W1OI-MjqQhULHL5e0diMpRHJZnYGqLTclo_qyBvPEU/s1600/gsm-rede_oi-1024x581.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9XxbYgHSu2kv-cIalLyC8DYW8LaqaUryD4qeS4CaQ6-yVihCK9J7Eel5VYnHHkRvylA6IvAl-faZtIOwgKpxFtp3uLuBrB_rU1W1OI-MjqQhULHL5e0diMpRHJZnYGqLTclo_qyBvPEU/s400/gsm-rede_oi-1024x581.png" width="400" /></a></div>
<br />
<br />
<h3>
<strong>Third attack, Neighborhood</strong></h3>
<div>
<strong><br /></strong></div>
<div style="text-align: justify;">
This last attack is the one with the best accuracy. The idea is based on the fact that a <strong>SMS</strong> or a <strong>call</strong> will be delivered faster between two members of the same BTS/Tower. In different towers the message will be delivered from one tower to another and so one, until it reaches the final destination. By having the two phones on the same BTS, the message will be delivered directly to the destination as soon as it hit the tower.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Similar to the second attack, the ID of the victim on the network should be revealed before anything. After that, the attack jumps from tower to tower verifying the amount of time that the network takes to send the <em>victim</em> to a proper time slot/channel to receive the message.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The coverage area of a BTS may vary, it tends to be bigger in rural areas, with lower phone density and smaller in downtown, bellow is illustrated the area of coverage from BTSs in my hometown. Where <strong>F</strong> and <strong>E</strong> are the biggest and the smallest coverage area. Its sizes can be found on the table bellow.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjd4KNoQo-8sH9FYYtrsxuedQXmWiIfhyijTg6-iVSbllcP_z0TCR3oB05cPBoxnaeTChvtHgun6y0Tpx5xg9JEi9qf3TQBeKG4fHSGD7oVkr-ZqC8EpeY5zF-C2RP2F0cFAT6e3Nv8LtI/s1600/gsm-rede_oi-bts-1024x566.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjd4KNoQo-8sH9FYYtrsxuedQXmWiIfhyijTg6-iVSbllcP_z0TCR3oB05cPBoxnaeTChvtHgun6y0Tpx5xg9JEi9qf3TQBeKG4fHSGD7oVkr-ZqC8EpeY5zF-C2RP2F0cFAT6e3Nv8LtI/s400/gsm-rede_oi-bts-1024x566.png" width="400" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtrEK0sAnXA-iteBhfcT0Zn68NFYUCNVRuJmZ0PAvS28U6-wGIpCT_SROAci4O8CNFeYVzetbOv3TbaMOuzmu6tGNRdY1ZyuIMdMPiUK6EyV68fwz5IsUEdwsGV4s8r8RaD-lqanQcOmU/s1600/table.png" imageanchor="1"><img border="0" height="82" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtrEK0sAnXA-iteBhfcT0Zn68NFYUCNVRuJmZ0PAvS28U6-wGIpCT_SROAci4O8CNFeYVzetbOv3TbaMOuzmu6tGNRdY1ZyuIMdMPiUK6EyV68fwz5IsUEdwsGV4s8r8RaD-lqanQcOmU/s400/table.png" width="400" /></a></div>
<br />
<h3>
<strong>Conclusion</strong></h3>
<div>
<strong><br /></strong></div>
<div style="text-align: justify;">
Those problems are not new, they have been discussed for a while. The idea of this post was to give short intro about the subject by exposing the vulnerabilities.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
All these hypothesis were proved on my dissertation. For further reading, including tests, have a look on my dissertation <a href="http://www.zimmerle.org/~zimmerle/master-final.pdf" target="_blank">available here</a>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For information about the ways to circumvent or minimize those problems, fire me an e-mail.</div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-54784461216190532982011-12-24T11:26:00.000-03:002013-09-10T14:27:31.258-03:00AppArmor D-Bus MediationsLooking like the SELinux but less boring, the AppArmor is a Linux security module (LSM) which provides mandatory access control (MAC). The first distro to adopt the utilization of AppArmor was SUSE in SUSE Linux Enterprise Server 10 and in openSUSE 10.1. It is part of Ubuntu since the version 8.04 and the adoption increase version to version since more profiles are created.<br />
<br />
Other software that is part of more and more applications each day is the D-Bus, adopted by GNOME and KDE as an inter-process communication mechanism, the usage of D-Bus allows the communication between different applications. It is used, for example, to provide the communication between a software Core with the UI. Due to the nature of the communication of certain applications (sensible data) is indispensable to have some control about who can acquire some interface or who can listen or send some message.<br />
<br />
D-Bus daemon has support to mediate SELinux messages and there is also a D-Bus internal mechanism that has some control over the use of the bus, but none of this is related to AppArmor. There are some experiments that show that it is possible however the necessary patches (Kernel, libapparmor and D-Bus daemon) were not submitted to be part of the respective projects, as explained in the earlier post.<br />
<br />
The patches on the experiment enable apparmor parser to understand the tag dbus, as illustrated on the example bellow (line 15). More information about the experiment and the syntax of the file can be seen in: <a href="https://lists.ubuntu.com/archives/apparmor/2011-September/001541.html">https://lists.ubuntu.com/archives/apparmor/2011-September/001541.html</a><br />
<br />
/home/zimmerle/hello.py flags=(complain) {<br />
#include <abstractions/base><br />
<br />
/usr/bin/python2.7 ix,<br />
/usr/include/python2.7/pyconfig.h r,<br />
/usr/local/lib/python2.7/dist-packages/ r,<br />
/usr/share/pyshared/PIL.pth r,<br />
/usr/share/pyshared/lazr.restfulclient-0.11.2-nspkg.pth r,<br />
/usr/share/pyshared/lazr.uri-1.0.2-nspkg.pth r,<br />
/usr/share/pyshared/pygst.pth r,<br />
/usr/share/pyshared/pygtk.pth r,<br />
/usr/share/pyshared/ubuntu-sso-client.pth r,<br />
/usr/share/pyshared/ubuntuone-client.pth r,<br />
<br />
dbus bar.foo.hello acquire,<br />
}<br />
<br />
In order to ensure the functionality of the suggestion made in the post: D-Bus Loadable security module support, I decided to modify the AppArmor D-Bus daemon patches to make them compatible with the suggested model. And it is working like a charm.<br />
<br />
The code of the current experiment can be fetched from:<br />
<br />
<a href="http://cgit.collabora.com/git/user/zimmerle/dbus-apparmor-lsm.git/">http://cgit.collabora.com/git/user/zimmerle/dbus-apparmor-lsm.git/</a><br />
<br />
Note that in this experiment I had to use the D-Bus internal functions/headers. I made little hacks in order to get it working but apparently, this is a good way to go.Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-50829980615027246842011-12-23T11:25:00.000-03:002013-09-10T14:27:31.255-03:00D-Bus Loadable security module supportWhile I was thinking about LSM mediations of the D-Bus messages, I found out a nice work that is being developed by the Ubuntu sec team in order to support the AppArmor mediation on D-Bus message exchange and service acquisition.<br />
<br />
Having a chat with John Johansen (from Unbuntu sec team), he said that he was missing a loadable module support on the D-Bus. Allowing the support of different Linux Security Modules mediation without messing up the D-Bus daemon code, which does make sense.<br />
<br />
I started to implement a little PoC about this loadable support, which consists in the following: the LSM modules can be dynamically loadable at the d-bus daemon startup. By copying a D-Bus LMS module to a given directory (which can be specified at the d-bus configuration) it will be loaded and registered.<br />
<br />
The idea is to have independent modules, if possible use only the D-Bus functions provided by libdbus, however, of course, if needed symbols can be copied from libdbus-internal.a.<br />
<br />
Despite the fact that the modules can be independent of the D-Bus internals, they must have at least one known function, this function should be named as “<em>pre_init</em>“, and receives the pointer to the D-Bus internal function “<em>register_security</em>“. The “<em>register_security</em>” function should be called by the module if it is loaded successfully. The “<em>pre_init</em>” function must return a “<em>dbus_bool_t</em>“: true if everything goes right or false if not. Note that audit can be also initialized by this function.<br />
<br />
The function “<em>register_security</em>” receives as parameter a pointer to the structure “<em>security_validations</em>” that is part of dbus-security.h. The structure is illustrated bellow:<br />
<pre class="prettyprint linenums">struct security_validations
{
char *name;
dbus_bool_t (*bus_security_allows_send) (DBusConnection *,
DBusConnection*,
const char *,
const char *,
const char *,
const char *,
const char *,
const char *,
const char *,
DBusError *);
dbus_bool_t (*bus_security_allows_acquire_service) (DBusConnection *,
const char *,
const char *,
DBusError *);
dbus_bool_t (*shutdown) (void);
};
</pre>
<br />
The structure “<em>security_validations</em>” defines the hooks and the name of the security module and also the function to shutdown the mediation. Two main hooks were needed, the first is the one responsible to mediate the message exchanges and the second is the responsible to avoid unauthorized process to acquire some service. The shutdown hook is not less important, but less used. Shutdown is only called when the D-Bus daemon is hanging out.<br />
<br />
The current implementation of SELinux mediation needs more hooks to work than what I am offering in this PoC. Since the SELinux implementation has some performance improvements by doing caching, it will be necessary to create new hooks to gather some information before deciding whether some message is ok to go or not, but this may be a later discussion.<br />
<br />
The patched D-Bus code is available at:<br />
<br />
<a href="http://cgit.collabora.com/git/user/zimmerle/dbus-lsm.git/">http://cgit.collabora.com/git/user/zimmerle/dbus-lsm.git/</a><br />
<br />
And there is a <em>dummy</em> module at:<br />
<br />
<a href="http://cgit.collabora.com/git/user/zimmerle/dbus-dummy-lsm.git/">http://cgit.collabora.com/git/user/zimmerle/dbus-dummy-lsm.git/</a>Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-32204995629211271062010-08-16T23:11:00.000-03:002013-09-10T14:27:31.249-03:00No eXecute and Atom, the current MeeGo stateThe security of your box goes over the firewall, or the fact that you are running a platform where all binaries are trusted. Even on that case, is still possible that vulnerabilities of some software or library could be exploited by a malicious party.<br />
<br />
The idea behind the NX bit, No eXecute, is to segregate the areas of the memory in two (lets keep it simple :P) big sets, the code execution area and the storage area. According to Wikipedia (<a href="http://en.wikipedia.org/wiki/List_of_Intel_Atom_microprocessors">http://en.wikipedia.org/wiki/List_of_Intel_Atom_microprocessors</a>), the Atom family has the capability to handle such bit.<br />
<br />
Adding this feature and a Linux kernel, is possible to avoid the execution of code in the data area, protecting the system against buffer overflows attack. However some marks should be placed on ELFs to archive such protection, these marks are made in the ELF construction and they can mark the ELF to have or not an executable stack. In the second case the executable flag has no effect, is useless.<br />
<br />
The marking can also be made on a library (it is also an ELF, duh!) and when this happens, the software which loads that library will be also allowed to run code inside the data segment, disabling again the protection against buffer overflow.<br />
<br />
To check the executable marks of your ELFs, you can use the pax-utils (<a href="http://www.gentoo.org/proj/en/hardened/pax-utils.xml">http://www.gentoo.org/proj/en/hardened/pax-utils.xml</a>). Running the tests on a daily MeeGo image (2010-22-07) the following results were archived:<br />
<br />
<code><br />[root@localhost ~]# scanelf -lpqeR<br />RWX --- --- /usr/lib/libmono.so.0.0.0<br />RWX --- --- /usr/lib/paxtest/getmain2<br />RWX --- --- /usr/lib/paxtest/getheap2<br />RWX --- --- /usr/bin/mono</code><br />
<br />
This means that libmono and mono, for some reason, are expected to run code on the data segment of the memory. In Fedora the mono is marked as RW, I dunno why it is marked as RWX in MeeGo, further investigation should be done.<br />
<br />
Mono’s GNU_STACK on Fedora:<br />
<code><br />(zimmerle@burbs)-(~/core/meego)$ readelf -l /usr/bin/mono | grep GNU_STACK<br />GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000</code><br />
<br />
Is acceptable to have some process without such kind of protecting, for example Java. Java depends on the executable stack to work. It is also acceptable to have some other binaries like: getmain2 and getheap2. These are used to test if the Machine is handling well the NX bit.<br />
<br />
To check if your platform has handled well the support of the NX bit, you can use the pax-test, really nice utility that allows us to check the protection against various kinds of exploration. Tests were also made on the same release used above.<br />
<br />
<strong>kidde mode:</strong><br />
<code><br />PaXtest - Copyright(c) 2003,2004 by Peter Busser <peter@adamantix.org>Released under the GNU Public Licence version 2 or later</code><br />
<br />
Writing output to paxtest.log<br />
It may take a while for the tests to complete<br />
Test results:<br />
PaXtest - Copyright(c) 2003,2004 by Peter Busser <peter@adamantix.org>Released under the GNU Public Licence version 2 or later<br />
<br />
Mode: kiddie<br />
Linux localhost.localdomain 2.6.35~rc6-131.2-netbook #1 SMP PREEMPT Tue Jul 27 14:34:50 UTC 2010 i686 i686 i386 GNU/Linux<br />
<br />
<code>Executable anonymous mapping : Killed<br />Executable bss : Killed<br />Executable data : Killed<br />Executable heap : Killed<br />Executable stack : Killed<br />Executable anonymous mapping (mprotect) : Vulnerable<br />Executable bss (mprotect) : Vulnerable<br />Executable data (mprotect) : Vulnerable<br />Executable heap (mprotect) : Vulnerable<br />Executable shared library bss (mprotect) : Vulnerable<br />Executable shared library data (mprotect): Vulnerable<br />Executable stack (mprotect) : Vulnerable<br />Anonymous mapping randomisation test : 12 bits (guessed)<br />Heap randomisation test (ET_EXEC) : 13 bits (guessed)<br />Heap randomisation test (ET_DYN) : 16 bits (guessed)<br />Main executable randomisation (ET_EXEC) : No randomisation<br />Main executable randomisation (ET_DYN) : 10 bits (guessed)<br />Shared library randomisation test : No randomisation<br />Stack randomisation test (SEGMEXEC) : 19 bits (guessed)<br />Stack randomisation test (PAGEEXEC) : 19 bits (guessed)<br />Return to function (strcpy) : Vulnerable<br />Return to function (strcpy, RANDEXEC) : Vulnerable<br />Return to function (memcpy) : Vulnerable<br />Return to function (memcpy, RANDEXEC) : Vulnerable<br />Executable shared library bss : Killed<br />Executable shared library data : Killed<br />Writable text segments : Vulnerable</code><br />
<br />
<strong>blackhat mode:</strong><br />
<code><br />PaXtest - Copyright(c) 2003,2004 by Peter Busser <peter@adamantix.org>Released under the GNU Public Licence version 2 or later</code><br />
<br />
Writing output to paxtest.log<br />
It may take a while for the tests to complete<br />
Test results:<br />
PaXtest - Copyright(c) 2003,2004 by Peter Busser <peter@adamantix.org>Released under the GNU Public Licence version 2 or later<br />
<br />
Mode: blackhat<br />
Linux localhost.localdomain 2.6.35~rc6-131.2-netbook #1 SMP PREEMPT Tue Jul 27 14:34:50 UTC 2010 i686 i686 i386 GNU/Linux<br />
<br />
<code>Executable anonymous mapping : Killed<br />Executable bss : Killed<br />Executable data : Killed<br />Executable heap : Killed<br />Executable stack : Killed<br />Executable anonymous mapping (mprotect) : Vulnerable<br />Executable bss (mprotect) : Vulnerable<br />Executable data (mprotect) : Vulnerable<br />Executable heap (mprotect) : Vulnerable<br />Executable shared library bss (mprotect) : Vulnerable<br />Executable shared library data (mprotect): Vulnerable<br />Executable stack (mprotect) : Vulnerable<br />Anonymous mapping randomisation test : 12 bits (guessed)<br />Heap randomisation test (ET_EXEC) : 13 bits (guessed)<br />Heap randomisation test (ET_DYN) : 16 bits (guessed)<br />Main executable randomisation (ET_EXEC) : No randomisation<br />Main executable randomisation (ET_DYN) : 10 bits (guessed)<br />Shared library randomisation test : No randomisation<br />Stack randomisation test (SEGMEXEC) : 19 bits (guessed)<br />Stack randomisation test (PAGEEXEC) : 19 bits (guessed)<br />Return to function (strcpy) : Vulnerable<br />Return to function (strcpy, RANDEXEC) : Vulnerable<br />Return to function (memcpy) : Vulnerable<br />Return to function (memcpy, RANDEXEC) : Vulnerable<br />Executable shared library bss : Killed<br />Executable shared library data : Killed<br />Writable text segments : Vulnerable</code><br />
<br />
As you can see, we are protected against code execution in any other area than that intended for this purpose. We don’t have randomization on libs due the fact that we are making use of the prelink, subject for another post .<br />
<br />
The pax-utils and pax-test pacakges can be found on my security MeeGo repostiory, at:<br />
<br />
<a href="http://meego.zimmerle.org/repo/security">http://meego.zimmerle.org/repo/security/</a><br />
<br />
If you are interested in testing it by yourself, you can download my ks file <a href="http://www.google.com/">here</a>.<br />
<br />
That kind of protection is very important almost mandatory, modern system still been hacked by such kind of attack class, when they opt to not provide such protection, the case of Xbox, for example which is exposed to a vulnerability in the 007: Agent Under Fire (http://en.wikipedia.org/wiki/Agent_Under_Fire_(video_game)).Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-3896441363985690412010-07-27T11:19:00.000-03:002016-12-26T22:40:44.074-03:00Poulsbo support on MeeGo, almost there<div style="text-align: justify;">
This is a continuation of my last post about Poulsbo support on MeeGo. A lot of people have asked me to continue this work, but I just had time today. In this post I will talk about the Xorg drivers. After some research/reading these sites:</div>
<ul><div style="text-align: justify;">
<br /></div>
<li><a href="http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/x11-driver-video-psb/">http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/x11-driver-video-psb/</a></li>
<br />
<li><a href="http://code.google.com/p/gma500/">http://code.google.com/p/gma500/</a></li>
<br />
<li><a href="http://www.happyassassin.net/2010/05/21/video-acceleration-and-poulsbo-news/">http://www.happyassassin.net/2010/05/21/video-acceleration-and-poulsbo-news/</a></li>
<br />
<li><a href="https://edge.launchpad.net/~gma500/+archive/fix">https://edge.launchpad.net/~gma500/+archive/fix</a></li>
</ul>
<br />
<div style="text-align: justify;">
And after some work, I created/built the necessary packages to get the Poulsbo Xorg driver working on MeeGo. The performance differences are notable.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I got my netbook running MeeGo with the Xorg driver, but without 3D acceleration yet, meaning, it still slow. The Intel closed 3D driver is already packed and installed but for an unknown reason when I enable it the Xorg just crash. Further investigation is needed.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I had to place “suid” bit on Xorg to make the driver work correctly. All the necessary packages to make it work can be found on my megoo repo, and also there is an image available if you want to check it out.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
You can download the image via torrent or directly from my site: <a href="http://meego.zimmerle.org/imgs/meego-core-xorg-psb-1.0.80.20100722.0045.iso">iso image</a>. The ks file goes <a href="http://meego.zimmerle.org/imgs/meego-core-xorg-psb-1.0.80.20100722.0045.ks">here</a>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And here is a nice picture of my netbook running the MeeGo official UI:</div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDIVltWpAI2IFlZBKVC66GiiHcRnicMqvy4vG-UKKsd-LyQOq3V2X1x2og_nhg6ggB6KEBaDZQr_e2X_zkVMR4dvNdwVD6-nIIRssthgVRkojGejGpDyaH3iP79ke-mHRSInHh6hIxAnE/s1600/sony_meego1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDIVltWpAI2IFlZBKVC66GiiHcRnicMqvy4vG-UKKsd-LyQOq3V2X1x2og_nhg6ggB6KEBaDZQr_e2X_zkVMR4dvNdwVD6-nIIRssthgVRkojGejGpDyaH3iP79ke-mHRSInHh6hIxAnE/s400/sony_meego1.jpg" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOxQaPST5VefQ_gdPiryHnR8iZA0Ii_icGFQCn2WpTSU-KPGLAvKCOO7YE0ClLmuBMt9XYObzCGmiTaay4UqjonXYmeImNbHIFO5V4u4kTw68m1r_sA3er8u0tGSF6gomu5Vqqy6oum6s/s1600/sony_meego2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOxQaPST5VefQ_gdPiryHnR8iZA0Ii_icGFQCn2WpTSU-KPGLAvKCOO7YE0ClLmuBMt9XYObzCGmiTaay4UqjonXYmeImNbHIFO5V4u4kTw68m1r_sA3er8u0tGSF6gomu5Vqqy6oum6s/s400/sony_meego2.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b>[Update 2016-12-26] </b>Mandriva's svn repositories are no longer available. For further information check (external link): http://www.whoishostingthis.com/resources/mandriva/</div>
<br />Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-8317140771635696902010-06-14T23:11:00.000-03:002013-09-10T14:27:31.264-03:00MeeGo: @SELinux on %packages.<div style="text-align: justify;">
I finally bought a netbook and since I am intending to use it with some work stuff (meaning data that requires confidentiality and integrity) I started to tuning my MeeGo to make it more protected before place my data</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To make it more protected I think that it is interesting to confine some, let’s say, “untrusted applications”. Which basically means more restrictive control over the processes. Usually I use GRSecurity for that but this time I am using SELinux. Since I am dealing with RPM and Fedora use to be a reference (at least for me) in the support to the SELinux, most of the specs files were copied from Fedora including the policy. The policy should be well refined to fit my needs, but it will be the subject of another post.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Supporting SELinux involves to support not only the kernel part of SELinux (kernel-selinux-netbook), but to support a huge number of packages as you can see bellow:</div>
<ul><br />
<li>selinux-policy-targeted</li>
<li>selinux-policy-doc</li>
<li>bwidget</li>
<li>selinux-policy</li>
<li>setools-libs-python</li>
<li>setools-libs</li>
<li>libsepol</li>
<li>kernel-selinux-netbook</li>
<li>libselinux-ruby</li>
<li>ustr-debug</li>
<li>policycoreutils</li>
<li>libprelude-python</li>
<li>libprelude-perl</li>
<li>policycoreutils-python</li>
<li>pax-utils</li>
<li>audispd-plugins</li>
<li>libselinux</li>
<li>perf</li>
<li>policycoreutils-newrole</li>
<li>libprelude-ruby</li>
<li>checkpolicy</li>
<li>ustr-debug-static</li>
<li>audit-libs-python</li>
<li>libsemanage-static</li>
<li>setools</li>
<li>libsemanage-python</li>
<li>ustr</li>
<li>libselinux-static</li>
<li>audit-libs</li>
<li>libsemanage</li>
<li>setools-libs-tcl</li>
<li>libsepol-static</li>
<li>setools-console</li>
<li>libselinux-python</li>
<li>ustr-static</li>
<li>libprelude</li>
<li>libselinux-utils</li>
<li>audit</li>
</ul>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Part of these packages are not needed to make the SELinux work, but they are used by auxiliary applications which make SELinux easy to deal with. As you can see, these packages provide dependencies on Ruby, Perl and Python for example. I think we just need the python dependency. The big difference between my packages and Fedora’s packages is the fact that I refuse myself to port the Java SELinux utilities.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
All the support to that packages (and also the devel version of them) are available at my MeeGo repo at:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<a href="http://meego.zimmerle.org/repo/security/packages/">http://meego.zimmerle.org/repo/security/packages/</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To add SELinux to your image, you just need to add to your .ks file, the following repo:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: left;">
repo --name=security --baseurl=http://meego.zimmerle.org/repo/security/packages/</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: justify;">
And you also need to place the SELinux package group in the package section:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
@SELinux</div>
<div style="text-align: justify;">
kernel-selinux-notebook</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: left;">
An example of a kick start file can be downloaded here: <a href="http://meego.zimmerle.org/repo/security/build/meego-netbook-chromium-ia32-security-1.0.20100614.1459.ks">http://meego.zimmerle.org/repo/security/build/meego-netbook-chromium-ia32-security-1.0.20100614.1459.ks</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: left;">
You can also download a SELinux MeeGo image at: <a href="http://meego.zimmerle.org/repo/security/build/">http://meego.zimmerle.org/repo/security/build/</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Here goes a picture of my netbook running selinux kernel:</div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgh-N-vXofG8MOXrRAvAleqkmSs0CiozQobqY19_ulFycogE8-mPtErvEKFPSfFg4lLHzphSIsdn_QtPCKExtwa3lEPQeX0MnhYxsCAbEPswNhsaQnwwJ3jL1f0_ZpB8bsEYXk_wlwXw_I/s1600/4702450804_1080c8e351.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgh-N-vXofG8MOXrRAvAleqkmSs0CiozQobqY19_ulFycogE8-mPtErvEKFPSfFg4lLHzphSIsdn_QtPCKExtwa3lEPQeX0MnhYxsCAbEPswNhsaQnwwJ3jL1f0_ZpB8bsEYXk_wlwXw_I/s400/4702450804_1080c8e351.jpg" width="400" /></a></div>
<br />
<div style="text-align: justify;">
The policy is not loaded automatically after the boot and the file system is not labeled yet. To load the policy just use load_policy tool.</div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-30005408159023286262010-01-18T11:11:00.000-03:002013-09-10T14:27:31.247-03:004×4 inclinometer<div style="text-align: justify;">
For those who are interested in knowing how steep is your N900, or the object that supports it. Meet the 4×4 inclinometer.</div>
<br />
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/wrhcm3Yo-7k" width="420"></iframe>
</div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: justify;">
I developed it to use in my car, hence the name 4×4 inclinometer. Using this application I can know the slope of the obstacles or the ground below my car.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
According to the manual of the car, it can be in an angle of heel of 45 degrees with no problem, something higher than this is at my own risk. When I read this information, just imagined the software for the N900)</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The current version depends on Qt 4.6 with the animation framework. The animation is used to rotate the images of the car, smoothing the movement. I am not an expert in gimp, so forgive me for the images poorly done. Next version I will put a simple support for themes.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The intallation files are already in extras-devel, so you just need to apt-get it. And the sources are available at:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<a href="http://git.zimmerle.org/?p=inclinometer.git;a=summary">http://git.zimmerle.org/?p=inclinometer.git;a=summary</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The car image and the application background are Trademark of Troller Veiculos Especias S/A, <a href="http://www.troller.com.br/">http://www.troller.com.br</a></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-27704457643902840772010-01-07T11:11:00.000-03:002013-09-10T14:27:31.243-03:00Poulsbo @MeeGo<div style="text-align: justify;">
While I was trying to leave my MeeGo usable and secure, the need to put my video driver to work properly appeared because I was getting annoyed with the fact that I haven’t the “official” MeeGo Ui running yet.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The chipset is a Poulsbo. It is in the list of not supported hardwares on MeeGo (http://wiki.meego.com/Netbooks), but, somehow Mandriva and others distros make use of it, so I decide to take a look by my self.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The posts from Adam Williamson (http://www.happyassassin.net/2009/01/30/intel-gma-500-poulsbo-graphics-on-linux-a-precise-and-comprehensive-summary-as-to-why-youre-screwed/) were very useful and based on that I decided to take a look at Mandriva’s svn (http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/libdrm-psb/), just to try to port something that already exists to MeeGo platform.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Another good resource is: https://edge.launchpad.net/~gma500/+archive/fix</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
With all that information I started to port the packages to MeeGo, creating the rpm specs in order to generate the packages. I did not have time to finish all packages yet, the Xorg driver is still missing. The kernel driver and others required packages are available on my MeeGo repo. It means good framebuffer screen and cool Xfce session, but not MeeGo UI yet.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
A new Kernel is required to be installed since happened a conflict or something like that with another module which was compiled built-in in the official Kernel. As I said, the Xorg driver is still missing. I will work on that as soon as I find some time to do it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The packages are available on my MeeGo repo, at: http://meego.zimmerle.org/repo/psb/packages/</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The packages are:</div>
<ul>
<li>psb-firmware</li>
<li>psb-kernel-modules</li>
<li>kernel-netbook-psb</li>
<li>psb-kernel-source</li>
<li>kernel-netbook-psb-devel</li>
</ul>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-27785688390656859782010-01-01T23:11:00.001-03:002013-09-10T14:27:31.245-03:00tcpdump && lipcap on extras-devel<div style="text-align: justify;">
For those who are playing with Maemo and network, now are available at Maemo extras-devel {fremantle|diablo} the tcpdump package and its dependency (libpcap) working like a charm.</div>
<br />
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/GEQh6SbULLw" width="420"></iframe></div>
<div style="text-align: center;">
<br /></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-22831080115617329882010-01-01T23:11:00.000-03:002013-09-10T14:27:31.240-03:00Iptables on extras devel<div style="text-align: justify;">
The iptables package is on Maemo extras devel. There is no support for connection state on the device Kernel consequently the NAT is not working. I tried to compile the modules, but I found myself in trouble trying to load them at the device. If you want to flash a kernel with support for connection state there is one available at my personal repository (read: mWall :: netfilter + ui for maemo for more information). Another discussion about that modules can be found: <a href="http://forums.internettablettalk.com/showthread.php?t=30916&page=1" target="_blank">here</a>.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkb8gKLp_yqVuz6F3kK2RFHECY_gxM3s3LDPE6nRZN7EQhVRNKTwGZGHuZSQKcKCtFKYVBpu8c_74vRJ3IJ7vUfyuFK7SkO1BsVAIpsKQ7qAav_YdKcoEX7Tx-6nRssL47rHv1FDaVTSk/s1600/screenshot05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkb8gKLp_yqVuz6F3kK2RFHECY_gxM3s3LDPE6nRZN7EQhVRNKTwGZGHuZSQKcKCtFKYVBpu8c_74vRJ3IJ7vUfyuFK7SkO1BsVAIpsKQ7qAav_YdKcoEX7Tx-6nRssL47rHv1FDaVTSk/s400/screenshot05.png" width="400" /></a></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-42018267200127841882009-12-21T23:11:00.000-03:002013-09-10T14:27:31.253-03:00mWall :: netfilter + ui for maemo<div style="text-align: justify;">
Something that certainly bothers me is the fact that i am always online independent of the network. I walk with my n900 in the pocket and sometimes I am using 3g, sometimes using wifi. I am jumping from trusted to untrusted wifi spots, and I have the strange feeling that maybe once (or more…) I will be part of a honeypot, malicious network or something like that.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As part of this type of network my device can be easily identified as an N900. (e.g. MAC address). Once the device is identified a person or a malicious software can start to guess passwords (rootme?) and can try to exploit softwares that are under development.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Avoiding been hacked on that situation I decided to write a small firewall UI for the n900 (netfilter/iptables back end), that allows me to block any incoming connection that is not authorized.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5saGdVqMX4SRn6AowjCl0QyypTEVFQnnLmtcsjxEvMmXQuVaKEv56FVXC2CzJ60X4EFphKuAGwExVsNLLlsXYwx9SUScNwvn8DmEGmGy3dmNDKxJ0Gl8Qwf4Zr8Ig4bTsGmicf9PBNKs/s1600/screenshot021.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5saGdVqMX4SRn6AowjCl0QyypTEVFQnnLmtcsjxEvMmXQuVaKEv56FVXC2CzJ60X4EFphKuAGwExVsNLLlsXYwx9SUScNwvn8DmEGmGy3dmNDKxJ0Gl8Qwf4Zr8Ig4bTsGmicf9PBNKs/s400/screenshot021.jpg" width="400" /></a></div>
<br />
<div style="text-align: justify;">
This is just a very first version of the firewall, a lot to be done yet. To install it on your device, check for mWall at my personal repository.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
You can install my repository by clicking here: <a href="http://maemo.zimmerle.org/zimmerle.install">zimmerle’s repo</a>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I also provide in my repository: the iptables package and a kernel with support to iptables state match. The iptables binary was marked with the suid bit, allowing its execution by users without super powers. But this should be fixed in the next release.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Let me advise you that the firewall rules are not permanent, I mean, you need to run the firewall in every boot. It is under development.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The code is available at: <a href="http://git.zimmerle.org/">http://git.zimmerle.org</a></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com1tag:blogger.com,1999:blog-257078517913940865.post-17778454376673367252009-01-27T23:11:00.000-03:002013-09-10T14:27:31.238-03:00Playing with Perl :P<div style="text-align: justify;">
I always had problems in reading my RSS feeds because I never found any good RSS reader (Google things are not an option here). The fact is: I’m not able to read my friends – or enemies – feeds, but I'm a good email reader so I decided to search a way to send the feeds to my email so, I will be able to read them.</div>
<br />
<div style="text-align: justify;">
I found a nice toy to do that, it is called: rss2email. Really nice toy. I just placed it to run on my server and it started to deliver my content.</div>
<br />
<div style="text-align: justify;">
After one week using it, I saw that I didn't subscribe any new feed. Imagine yourself logging in a server to add a new RSS feed... nah too complicated. Too much work for me.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So I decided to create a new mail alias to receive commands and process them by its procmail rules. Well ok, what about the security? And if that MTFK friend decide to clean up my entire feeds? That is why I decided to verify my gpg signature before process the commands and for that a combination of procmail rules and my cute Perl script is amazing.</div>
<div style="text-align: justify;">
<br /></div>
Here goes my Perl script and enjoy:<br />
<pre class="prettyprint linenums">
#!/usr/bin/perl
#
# Copyright (C) 2009 Felipe Zimmerle
#
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
#
# Config.
$ Config {'trusted'} = 'E8B11277';
$config{'debug'} = 1;
$config{'rss2email'} = '/your/path/to/rss2email/rss2email.py';
$config{'dat'} = '/your/path/to/rss2email/feeds.dat';
$config{'mailto'} = 'your@mail.something';
$config{'sendmail'} = '/usr/sbin/sendmail';
$config{'label'} = '/your/path/to/labels.txt';
$config{'mail'} =<<EOT;
To: $ config {'mailto'}
From: RMO Report <felipe-rss\@zimmerle.org>
Subject: rmo report
RMO Results:
RESULTS
EOT
# Do not edit above this line.
use IPC::Open3;
use IO::Handle;
use Encode;
undef $/;
my $mail = <>;
$mail =~ s/=(\n|\r|\n\r|\r\n)//gom;
$mail =~ s/=3D/=/gom;
$mail =~ s/=20/ /gom;
my ($ IN, $ OUT, $ ERR) = (IO :: Handle-> new (), IO :: Handle-> new (),
IO :: Handle-> new ());
open3 ($ IN, $ OUT, $ ERR "gpg") | | die "Unable to run: $ \ n";
print $IN $mail;
close($IN);
my $from,@cmd,$results,$pr,$ops;
# Parser the commands.
my $o = <$OUT>;
$o = ~ s / ^ add (\ "[A-z0-9_. -] + \" | [A-z0-9_. -] +) (. *) / @ {[Eval {$ pr + +; $ cmd [@ cmd] =
["add", "$2", "@{[eval {$a = $1; $a =~ s@^(\")(.*)(\"$)@$2@; $a;}]}"] }]}/gome;
$o = ~ s / ^ (the | delete) ([0-9] +) / @ {[eval {$ pr + +; $ cmd [@ cmd] = ["delete", "$ 2"] if $ pr 1 }]}/gome;
$o = ~ s / ^ list / @ {[$ cmd [@ cmd] = ["list"]]} / gome;
close($OUT);
# Check signature.
my $ e = <$ ERR>;
$from = $1 if $e =~ m/.*key ID ([A-z0-9]+)(\n|\r|\n\r|\r\n)gpg: Good signature from.*/m;
close($ERR);
die "Wow a hacker:\n$mail" if $from ne $config{'trusted'};
foreach my $c (@cmd) {
my $ config = $ c_ {'rss2email'}. "". $ Config {'dat'}.
" " . $c->[0] . " " . $c->[1];
$results .= "Command: $c_\n" . `$c_` . "\n";
`echo $c->[2],$c->[1] >> $config{'label'}` if $c->[0] eq "add";
}
$results .= "\nDelete request is just welcome if its come " .
" alone. $ops delete" .
"s"?"":$ops>1 . " ignored.\n" if ($ops > 0);
$config{'mail'} =~ s/RESULTS/$results/;
open (S, "| $ config {'sendmail'} t $ config {'mailto'}") or die "."
"Unable to run sendmail: $!\n";
print S $config{'mail'};
close(S);</pre>
<br />
<div style="text-align: justify;">
I really love Perl. Amazing, just few lines and my problem was solved ;P</div>Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-73864993970896152422008-03-23T11:11:00.000-03:002013-09-10T14:27:31.262-03:00WUSB Cable AssociationThis document explains the Cable Association (cba) between a MS Windows host and a IOGear HUB [1] using the WiMedia application [2]. The logs analyzed here have been generated by a usb sniffer, usbsnoop [3]. This analysis was made to clarify the cable association specification. It’s the result of one day's work by me and my friend Alex.<br />
<br />
According to the specifications something like that should happens:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhV9Jtp0GzRPnTBn64UIRFaUg2GYNzAbtpg9WKlFf2HXvMWrAm5yoT8f_t0q7K0c66ERceBF0HlWBNCMpO_S5NiEZVZFq64Z8Ktj7pYn7JKQliw9RZZ6Ndc5s_UeUQo0uEWW54T7fSx518/s1600/iogear_association_final.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhV9Jtp0GzRPnTBn64UIRFaUg2GYNzAbtpg9WKlFf2HXvMWrAm5yoT8f_t0q7K0c66ERceBF0HlWBNCMpO_S5NiEZVZFq64Z8Ktj7pYn7JKQliw9RZZ6Ndc5s_UeUQo0uEWW54T7fSx518/s1600/iogear_association_final.png" /></a></div>
<br />
Our analysis is divided into blocks. Each block has a USB control message. They are:<br />
<ul>
<li><a href="http://www.blogger.com/blogger.g?blogID=257078517913940865#um">Association information</a></li>
<li><a href="http://www.blogger.com/blogger.g?blogID=257078517913940865#dois">Host information</a></li>
<li><a href="http://www.blogger.com/blogger.g?blogID=257078517913940865#tres">Association request</a></li>
<li><a href="http://www.blogger.com/blogger.g?blogID=257078517913940865#quatro">Second association request</a></li>
<li><a href="http://www.blogger.com/blogger.g?blogID=257078517913940865#cinco">Setting association request</a></li>
</ul>
<div>
<br /></div>
<h2>
ASSOCIATION INFORMATION</h2>
Our first usb_message_control is a request of information association [4].<br />
<br />
<pre style="background: #fffceb; border: 1px double #ddd8b9; padding-left: 3px; padding-right: 3px; padding-top: 10px;">[12 ms] >>> URB 5 going down >>>
-- URB_FUNCTION_CLASS_INTERFACE:
TransferFlags = 00000001 (USBD_TRANSFER_DIRECTION_IN, ~USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 00000100
TransferBuffer = 897ffd48
TransferBufferMDL = 00000000
UrbLink = 00000000
RequestTypeReservedBits = 00000001
Request = <span style="background: #ff5d5d; border: none; padding-left: 3px; padding-right: 3px;">00000001</span>
Value = 00000000
Index = 00000000
[12 ms] UsbSnoop - MyInternalIOCTLCompletion(bab39db0) : fido=00000000, Irp=896c8008, Context=897bd968, IRQL=2
[12 ms] <<< URB 5 coming back <<<
-- URB_FUNCTION_CONTROL_TRANSFER:
PipeHandle = 89923990
TransferFlags = 0000000b (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 00000019
TransferBuffer = 897ffd48
TransferBufferMDL = 8a37b4c0
UrbLink = 00000000
SetupPacket = 00000000: a1 01 00 00 00 00 00 01
</pre>
<br />
<div style="border-left: none; border-right: none; border-top: 0px solid #fff; color: #kkk; margin-left: 30px; margin-top: 4px; padding: 0px 0px 0px 0px;">
00000000: <span style="background: #FA8DFA; border: none; padding-left: 3px; padding-right: 3px;">19 00</span> <span style="background: #8EE4FD; border: none; padding-left: 3px; padding-right: 3px;">02</span> <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">00 00</span> <span style="border: 1px dotted #666666; padding: 3px;"><span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">01</span> <span style="background: #6666FF; border: none; padding-left: 3px; padding-right: 3px;">00</span> <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">01 00</span> <span style="background: #C6C6FF; border: none; padding-left: 3px; padding-right: 3px;">00 00</span> <span style="background: #FFFF66; border: none; padding-left: 3px; padding-right: 3px;">00 00 00 00</span></span> <span style="border-right: none; border: 1px dashed #666666; padding: 3px;"><span style="background: #B1B165; border: none; padding-left: 3px; padding-right: 3px;">02</span></span></div>
<br />
00000010: <span style="border: 1px dashed rgb(102, 102, 102); padding: 3px;"><span style="background: #B871FF; border: none; padding-left: 3px; padding-right: 3px;">00</span> <span style="background: #F15864; border: none; padding-left: 3px; padding-right: 3px;">01 00</span> <span style="background: #9DD297; border: none; padding-left: 3px; padding-right: 3px;">01 00</span> <span style="background: #8DC1D6; border: none; padding-left: 3px; padding-right: 3px;">6c 00 00 00</span></span><br />
<br />
<br />
<br />
<span style="padding-left: 4px;"><b>These colored bytes represent:</b></span><br />
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11px;">
<br />
<table border="0" cellpadding="4" cellspacing="0" style="valign: top;"><tbody>
<tr valign="top"><td width="90"></td><td><span style="padding-left: 4px;"><b>Request:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #ff5d5d; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00000001</span></td><td><span style="padding-left: 4px;">Represent a CBAF GET_ASSOCIATION_INFORMATION request</span></td></tr>
<tr valign="top"><td></td><td><span style="padding-left: 4px;"><b>Response:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FA8DFA; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">19 00</span></td><td><span style="padding-left: 4px;">The full size of this structurre (including this two bytes), in this case 25 (0×19) bytes</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #8EE4FD; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02</span></td><td><span style="padding-left: 4px;">Number of association requests, in this case: 2. See the arrays bellow</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #93FF64; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02 00</span></td><td><span style="padding-left: 4px;">Fixed value for this kind of association</span></td></tr>
<tr valign="top"><td></td><td><span style="padding-left: 4px;"><b>Association Requests Array:</b></span></td></tr>
<tr valign="top"><td></td><td><span style="padding-left: 4px;"><b>First array element:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFCC00; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01</span></td><td><span style="padding-left: 4px;">Index value. 01 in this case</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #6666FF; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00</span></td><td><span style="padding-left: 4px;">Reserved byte…</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFD2A6; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 00</span></td><td><span style="padding-left: 4px;">Certified Wireless USB should have value: 0×1 in this field.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #C6C6FF; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00 00</span></td><td><span style="padding-left: 4px;">00 00 == RetriveHostInfo, in this case the host should send its CHID value to the device, should happend before the AssociateWUSB, as happend here.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFFF66; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00 00 00 00</span></td><td><span style="padding-left: 4px;">Represent the size of the association type, zero in this case.</span></td></tr>
<tr valign="top"><td></td><td><span style="padding-left: 4px;"><b>Last array element:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #B1B165; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02</span></td><td><span style="padding-left: 4px;">Index of the array, 02 in this case</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #B871FF; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00</span></td><td><span style="padding-left: 4px;">Reserved byte…</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #F15864; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 00</span></td><td><span style="padding-left: 4px;">Certified Wireless USB should have value: 0×1 in this field.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #9DD297; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 00</span></td><td><span style="padding-left: 4px;">00 01 == AssociateWUSB, in this case the host will generate a response that contains the CC and return it to the device as the RetriveHostInfo it is mandatory in the association </span></td></tr>
<tr valign="top"><td align="right"><span style="background: #8DC1D6; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">6c 00 00 00</span></td><td><span style="padding-left: 4px;">Association type info size, in this case 108 bytes.</span></td></tr>
</tbody></table>
<br /></div>
<br />
In this block we have a GET_ASSICIATION_INFORMATION [4] request with an ASSOCIATION_INFORMATION [5] anwser. This ASSOCIATION_INFORMATION are composed by two ASSOCIATION_REQUEST [6], one is a RetriveHostInfo request and the another one is AssociateWUSB [7].<br />
<br />
<br />
<h2>
HOST INFORMATION</h2>
As a response of our first usb_message_control request, bellow is the host info.<br />
<pre style="background: #fffceb; border: 1px double #ddd8b9; padding-left: 3px; padding-right: 3px; padding-top: 10px;">[12 ms] >>> URB 6 going down >>>
-- URB_FUNCTION_CLASS_INTERFACE:
TransferFlags = 00000000 (USBD_TRANSFER_DIRECTION_OUT, ~USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 00000054
TransferBuffer = 8a365268
TransferBufferMDL = 00000000
UrbLink = 00000000
RequestTypeReservedBits = 00000001
Request = 00000003
Value = 00000101
Index = 00000000 [12 ms]
UsbSnoop - MyInternalIOCTLCompletion(bab39db0) : fido=00000000, Irp=896c8008, Context=89b253a8, IRQL=2 [12 ms]
URB 6 coming back
<<<
-- URB_FUNCTION_CONTROL_TRANSFER:
PipeHandle = 89923990
TransferFlags = 0000000a (USBD_TRANSFER_DIRECTION_OUT, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 00000054
TransferBuffer = 8a365268
TransferBufferMDL = 8a37b4c0
UrbLink = 00000000
SetupPacket = 00000000: 21 03 01 01 00 00 54 00
</pre>
<br />
<div style="border-left: none; border-right: none; border-top: 0px solid #fff; color: #kkk; margin-left: 30px; margin-top: 4px; padding: 0px 0px 0px 0px;">
00000000: <span style="background: #FA8DFA; border: none; padding-left: 3px; padding-right: 3px;">00 00 02 00 01 00</span> <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">01 00 02 00 00 00</span> <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">00 10 10 00</span><br />
00000010: <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">13 c7 31 42 52 44 30 30 32 30 30 30 c4 9a d5 70</span><br />
00000020: <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">08 00 02 00 10 33</span> <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">0c 00 2a 00 57 00 69 00 43 00</span><br />
00000030: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">65 00 6e 00 74 00 65 00 72 00 20 00 57 00 69 00</span><br />
00000040: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">72 00 65 00 6c 00 65 00 73 00 73 00 20 00 55 00</span><br />
00000050: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">53 00 42 00</span><br />
<br />
<br />
<span style="padding-left: 4px;"><b>These colored bytes represent:</b></span><br />
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11px;">
<br />
<table border="0" cellpadding="4" cellspacing="0" style="valign: top;"><tbody>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>USB Control Message:</b></span></td></tr>
<tr valign="top"><td align="right" width=""120"><span style="background: #ff5d5d; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00000003</span></td><td><span style="padding-left: 4px;">SET_ASSOCIATION_RESPONSE</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FA8DFA; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00 00 02 00 01 00</span></td><td><span style="padding-left: 4px;">Association type id as expected, this values are filled with the values of the last request. Attribute id: 0x0, Attribute length: 0x2 and data 0x1.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #93FF64; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 00 02 00 00 00</span></td><td><span style="padding-left: 4px;">Association sub type id.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFCC00; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">13 c7 31 42 52 44 30 30 32 30 30 30 c4 9a d5 70</span></td><td><span style="padding-left: 4px;">CHID¹.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFD2A6; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">08 00 02 00 10 33</span></td><td><span style="padding-left: 4px;">Lang ID, Unicode language id code used in the next field.</span></td></tr>
<tr valign="top"><td align="right" width="120"><span style="background: #b3e4f4; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">0c 00 2a 00 57 00 69 00 43 00 65 00 6e 00 74 00 65 00 72 00 20 00 57 00 69 00 72 00 65 00 6c 00 65 00 73 00 73 00 20 00 55 00 53 00 42 00</span></td><td><span style="padding-left: 4px;">Host friendly name, in unicode form. In this case: "WiCenter Wireless USB" ( \x57\x69\x43\x65\x6e\x74\x65\x72\x20\x57\x69\x72\x65\x6c\x65\x73\x73\x20\x55\x53\x42 )</span></td></tr>
</tbody></table>
<br /></div>
<br />
This is block is structured as a HOST_INFO [8]. Its request is SET_ASSOCIATION_RESPONSE.<br />
<br />
<a href="http://www.blogger.com/blogger.g?blogID=257078517913940865" name="tres"></a><br />
<h2>
ASSOCIATION REQUEST</h2>
<br />
As requested at the first data exchange, this is the information about the device:<br />
<pre style="background: #fffceb; border: 1px double #ddd8b9; padding-left: 3px; padding-right: 3px; padding-top: 10px;">[12 ms] >>> URB 7 going down >>>
-- URB_FUNCTION_CLASS_INTERFACE:
TransferFlags = 00000003 (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 0000002c
TransferBuffer = 8a4c50a8
TransferBufferMDL = 00000000
UrbLink = 00000000
RequestTypeReservedBits = 00000001
Request = <span style="background: #ff5d5d; border: none; padding-left: 3px; padding-right: 3px;">00000002</span>
Value = 00000200
Index = 00000000
[13 ms] UsbSnoop - MyInternalIOCTLCompletion(bab39db0) : fido=00000000, Irp=896c8008, Context=8a3c0f58, IRQL=2
[13 ms] <<< URB 7 coming back <<<
-- URB_FUNCTION_CONTROL_TRANSFER:
PipeHandle = 89923990
TransferFlags = 0000000b (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 0000002c
TransferBuffer = 8a4c50a8
TransferBufferMDL = 8a37b4c0
00000000: <span style="background: #FA8DFA; border: none; padding-left: 3px; padding-right: 3px;">02 00 04 00 6c 00 00 00</span> <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">01 10 10 00 2a 5e 70 14</span>
00000010: <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">ab 74 ec 49 e1 59 15 03 ee f6 f9 6c </span> <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">04 10 02 00</span>
00000020: <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">01 00</span> <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">08 00 02 00 09 04</span> <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">0b 00 40 00</span>
UrbLink = 00000000
SetupPacket =
00000000: a1 02 00 02 00 00 2c 00</pre>
<br />
<span style="padding-left: 4px;"><b>These colored bytes represent:</b></span><br />
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11px;">
<table border="0" cellpadding="4" cellspacing="0" style="valign: top;"><tbody>
<tr valign="top"><td width="120"></td><td></td></tr>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>Request:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #ff5d5d; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00000002</span></td><td><span style="padding-left: 4px;">GET_ASSOCIATION_REQUEST</span></td></tr>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>Response:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FA8DFA; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02 00 04 00 6c 00 00 00</span></td><td><span style="padding-left: 4px;">Size of this structure. 0x2 represents the attribute type id and 0x4 represents the attribute length. 108 bytes in this case</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #93FF64; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 10 10 00 2a 5e 70 14 ab 74 ec 49 e1 59 15 03 ee f6 f9 6c</span></td><td><span style="padding-left: 4px;">CDID</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFCC00; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">04 10 02 00 01 00</span></td><td><span style="padding-left: 4px;">The last 4 bytes is the band group. See section 7.4.1 of WUSB specification</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFD2A6; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">08 00 02 00 09 04</span></td><td><span style="padding-left: 4px;">Language ID, used by the next field.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #b3e4f4; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">0b 00 40 00</span></td><td><span style="padding-left: 4px;">Device friendly name, in unicode format.</span></td></tr>
</tbody></table>
<br /></div>
<br />
This block contains the information about the device, it exchange information about the device id, supported band groups and device friendly name [9].<br />
<br />
<a href="http://www.blogger.com/blogger.g?blogID=257078517913940865" name="quatro"></a><br />
<h2>
SECOND ASSOCIATION REQUEST</h2>
The same request as above but now the response is complete with the device friendly name.<br />
<pre style="background: #fffceb; border: 1px double #ddd8b9; padding-left: 3px; padding-right: 3px; padding-top: 10px;">[13 ms] >>> URB 8 going down >>>
-- URB_FUNCTION_CLASS_INTERFACE:
TransferFlags = 00000003 (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 0000006c
TransferBuffer = 89ae84c0
TransferBufferMDL = 00000000
UrbLink = 00000000
RequestTypeReservedBits = 00000001
Request = <span style="background: #ff5d5d; border: none; padding-left: 3px; padding-right: 3px;">00000002</span>
Value = 00000200
Index = 00000000
[13 ms] UsbSnoop - MyInternalIOCTLCompletion(bab39db0) : fido=00000000, Irp=896c8008, Context=8a573930, IRQL=2
[13 ms] <<< URB 8 coming back <<<
-- URB_FUNCTION_CONTROL_TRANSFER:
PipeHandle = 89923990
TransferFlags = 0000000b (USBD_TRANSFER_DIRECTION_IN, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 0000006c
TransferBuffer = 89ae84c0
TransferBufferMDL = 8a37b4c0
00000000: <span style="background: #FA8DFA; border: none; padding-left: 3px; padding-right: 3px;">02 00 04 00 6c 00 00 00</span> <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">01 10 10 00 2a 5e 70 14</span>
00000010: <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">ab 74 ec 49 e1 59 15 03 ee f6 f9 6c</span> <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">04 10 02 00</span>
00000020: <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">01 00</span> <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">08 00 02 00 09 04</span> <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">0b 00 40 00 49 00 4f 00</span>
00000030: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">47 00 45 00 41 00 52 00 20 00 57 00 55 00 53 00</span>
00000040: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">42 00 20 00 48 00 75 00 62 00 00 00 00 00 00 00</span>
00000050: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span>
00000060: <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">00 00 00 00 00 00 00 00 00 00 00 00</span>
UrbLink = 00000000
SetupPacket =
00000000: a1 02 00 02 00 00 6c 00</pre>
<br />
<span style="padding-left: 4px;"><b>These colored bytes represent:</b></span><br />
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11px;">
<br />
<table border="0" cellpadding="4" cellspacing="0" style="valign: top;"><tbody>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>Request:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #ff5d5d; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00000002</span></td><td><span style="padding-left: 4px;">GET_ASSOCIATION_REQUEST</span></td></tr>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>Response:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FA8DFA; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02 00 04 00 6c 00 00 00</span></td><td><span style="padding-left: 4px;">Size of this structure. 108 bytes in this case</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #93FF64; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 10 10 00 2a 5e 70 14 ab 74 ec 49 e1 59 15 03 ee f6 f9 6c</span></td><td><span style="padding-left: 4px;">CDID</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFCC00; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">04 10 02 00 01 00</span></td><td><span style="padding-left: 4px;">Group band. See section 7.4.1 of WUSB specification.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFD2A6; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">08 00 02 00 09 04</span></td><td><span style="padding-left: 4px;">Language ID, used by the next field.</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #b3e4f4; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;"><span style="background: #b3e4f4; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">0b 00 40 00 49 00 4f 00 47 00 45 00 41 00 52 00 20 00 57 00 55 00 53 00 42 00 20 00 48 00 75 00 62 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span></span></td><td><span style="padding-left: 4px;">Device friendly name, in unicode format. In this case: "IOGEAR WUSB Hub" ( \x49\x4f\x47\x45\x41\x52\x20\x57\x55\x53\x42\x20\x48\x75\x62 )</span></td></tr>
</tbody></table>
<br /></div>
<br />
<a href="http://www.blogger.com/blogger.g?blogID=257078517913940865" name="cinco"></a><br />
<h2>
SETTING ASSOCIATION REQUEST</h2>
<br />
This block communicates the success of the cable association operation [10]. If the association is not succeded an AssociationStatus (Attr id: 0x4 and Attr length: 0x4) is expected with the reason. The reason should be one of the listed bellow:<br />
<ul><br />
<li>0x1, Association unsuccessful</li>
<li>0x2, Malformated association request</li>
<li>0x3, Association type not supported</li>
</ul>
<br />
<pre style="background: #fffceb; border: 1px double #ddd8b9; padding-left: 3px; padding-right: 3px; padding-top: 10px;">[10589 ms] >>> URB 9 going down >>>
-- URB_FUNCTION_CLASS_INTERFACE:
TransferFlags = 00000000 (USBD_TRANSFER_DIRECTION_OUT, ~USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 0000004e
TransferBuffer = 8a003530
TransferBufferMDL = 00000000
00000000: <span style="background: #FA8DFA; border: none; padding-left: 3px; padding-right: 3px;">00 00 02 00 01 00</span> <span style="background: #93FF64; border: none; padding-left: 3px; padding-right: 3px;">01 00 02 00 01 00</span> <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">02 00 04 00</span>
00000010: <span style="background: #FFCC00; border: none; padding-left: 3px; padding-right: 3px;">4e 00 00 00</span> <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">02 10 30 00 13 c7 31 42 52 44 30 30</span>
00000020: <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">32 30 30 30 c4 9a d5 70 2a 5e 70 14 ab 74 ec 49</span>
00000030: <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">e1 59 15 03 ee f6 f9 6c d7 a6 f4 4c 6d 88 0f be</span>
00000040: <span style="background: #FFD2A6; border: none; padding-left: 3px; padding-right: 3px;">b6 0c 25 ef 6f 24 a3 ed</span> <span style="background: #b3e4f4; border: none; padding-left: 3px; padding-right: 3px;">04 10 02 00 01 00</span>
UrbLink = 00000000
RequestTypeReservedBits = 00000001
Request = <span style="background: #ff5d5d; border: none; padding-left: 3px; padding-right: 3px;">00000003</span>
Value = 00000201
Index = 00000000
[10623 ms] UsbSnoop - MyInternalIOCTLCompletion(bab39db0) : fido=00000000, Irp=896f0008, Context=896caa40, IRQL=2
[10623 ms] <<< URB 9 coming back <<<
-- URB_FUNCTION_CONTROL_TRANSFER:
PipeHandle = 89923990
TransferFlags = 0000000a (USBD_TRANSFER_DIRECTION_OUT, USBD_SHORT_TRANSFER_OK)
TransferBufferLength = 0000004e
TransferBuffer = 8a003530
TransferBufferMDL = 898aac20
UrbLink = 00000000
SetupPacket =
00000000: 21 03 01 02 00 00 4e 00</pre>
<br />
<span style="padding-left: 4px;"><b>These colored bytes represent:</b></span><br />
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 11px;">
<br />
<table border="0" cellpadding="4" cellspacing="0" style="valign: top;"><tbody>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>Request:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #ff5d5d; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00000003</span></td><td><span style="padding-left: 4px;">SET_ASSOCIATION_REQUEST</span></td></tr>
<tr valign="top"><td width="120"></td><td><span style="padding-left: 4px;"><b>Response:</b></span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FA8DFA; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">00 00 02 00 01 00</span></td><td><span style="padding-left: 4px;">Association Type ID</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #93FF64; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">01 00 02 00 01 00</span></td><td><span style="padding-left: 4px;">Association Sub Type ID</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFCC00; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02 00 04 00 4e 00 00 00</span></td><td><span style="padding-left: 4px;">Length of this data structure</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #FFD2A6; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">02 10 30 00 13 c7 31 42 52 44 30 30 32 30 30 30 c4 9a d5 70 2a 5e 70 14 ab 74 ec 49 e1 59 15 03 ee f6 f9 6c d7 a6 f4 4c 6d 88 0f be b6 0c 25 ef 6f 24 a3 ed</span></td><td><span style="padding-left: 4px;">CC²</span></td></tr>
<tr valign="top"><td align="right"><span style="background: #b3e4f4; border: 1px solid #CCCCCC; padding-left: 3px; padding-right: 3px;">04 10 02 00 01 00</span></td><td><span style="padding-left: 4px;">Band group</span></td></tr>
</tbody></table>
<br /></div>
<br />
<hr />
<br />
<br />
¹ CHID, Conection Host Identify<br />
² CC, Connection Context<br />
<br />
[1] http://www.iogear.com/product/GUWH104KIT/<br />
[2] http://www.iogear.com/support/dm/driver/GUWH104KIT#display<br />
[3] http://benoit.papillault.free.fr/usbsnoop/<br />
[4] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-1<br />
[5] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-3<br />
[6] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-4<br />
[7] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-5<br />
[8] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-7<br />
[9] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-8<br />
[10] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-9<br />
[11] Association Models Supplement to the Certified Wireless Universal Serial Bus Specification<br />
Revision 1.0 - Table 4-10<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0tag:blogger.com,1999:blog-257078517913940865.post-8755586826943904842006-12-25T23:11:00.000-03:002013-09-10T14:27:31.266-03:00Exchangeable image file format for Digital Still Cameras: Exif<div style="text-align: justify;">
After some google searches I concluded that doesn't exist a python library that’s able me to manipulate some data in a JPEG Exif. I need this, cause I’m involved in a project called <a href="http://syncropated.garage.maemo.org/" target="_blank" title="Syncropated">Syncropated</a>, and this software wants to embed a thumbnail in JPEGs files. As you can see at Exif_2-1_V1.PDF, section 2.5.5 it’s possible.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But if I will code something to write this thumbnail, why do not put some code to parse data and something else?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Below you can read how the Exif works and in another post I will talk more about the Python Exif library that I’m coding.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
First we want to know if the file is a JPEG or not, so we can simple check the two first bytes of the file. If the byte[0] equal then ‘FF’ and the second one is ‘D8′ (Figure 1, <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid2IDAOLjObWJmy9rEoQ8rexQVZJKUEmTiOP2pic1_qRvygWdTVE_np4_AFXzgGy6ZRk-vwR5p_BTeUi5DpZP-177VblQDf-YpZQWgDdJRhPK5yMJ4PJhQMWVURIbSqJzTO7mOtfntrHo/s1600/bnt_green.jpg" />) the file can be considered a JPEG candidate. To be a JPEG, it must follow others rules shown below. JPEG files that has Exif must have the word `Exif` in the header. To be more specific these words are supposed to start at the 6th byte of the file, forming a sequence like the green one (<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDzxtR975khNhf3T5nysAFZdUsa-aimWk7ceLU_Elx9k66Z-KoZCjrSjMXHd7pcHdxsP062MqjqbtgILMUkRXyu3miZy2mNyCBSPMbrHcFkKabBKlyN82hYzc0eZfjmtvOys-xJcLQA8M/s1600/bnt_really_green.jpg" />) one that you can see at <em>Figure 1</em>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The 49 49 (Figure 1, <img height="12" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjScGkS6fP08TXYqfpwuBA5YinQQXSc6AgMB5SbcJNdG3SkYkRv4Y82F0LPKjV_6xhhb4xfWb-MWHAs5aQYYRy8Wssnh_Bq0dMVS_l94JlRcbubBjGjJ1e8lTIZl0dSfHSweUZbA-DYQSo/s1600/bnt_pink.jpg" width="12" />) represent the ordering of the data sequences, if its is <a href="http://en.wikipedia.org/wiki/Endianness#Big-endian">Big-endian</a> or <a href="http://en.wikipedia.org/wiki/Endianness#Big-endian">Little-endian.</a> Little-endian is represented by `II` (Intel format) and Big-endian is `MM` (Motorola format).</div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxlZL9_J4CL1v_06l-wXWfLDop8kLWCuyc9JFDRQGTtFrSP8fjAtudYS3gksFJcI9e7klJaXEVZZXU1vy7viGjgMahfm_Jng04egBJEAmTN6NpPdZI1xgBTuiA5dDFhyphenhyphenhPt2rPN42hilU/s1600/exif-khexedit.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxlZL9_J4CL1v_06l-wXWfLDop8kLWCuyc9JFDRQGTtFrSP8fjAtudYS3gksFJcI9e7klJaXEVZZXU1vy7viGjgMahfm_Jng04egBJEAmTN6NpPdZI1xgBTuiA5dDFhyphenhyphenhPt2rPN42hilU/s1600/exif-khexedit.jpg" /></a></div>
<div style="text-align: center;">
Figure 1. JPEG File raw bytes.</div>
<br />
<br />
Other important thing to know about Exit is the organization of the data, Exif is divided in:<br />
<ul><br />
<li>JPEG HEADER</li>
<li>0th IFD</li>
<li>0th IFD Values</li>
<li>1st IFD</li>
<li>1st IFD Values</li>
<li>1st Thumbnail – Image Data</li>
<li>0th (Primary) – Image Data</li>
</ul>
<div style="text-align: justify;">
An IFD (Image File Directory) is used to store tags, with values and data types. IFDs works like a chained lists. The IF0 points to ID1 and so so…</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
All IFDs have the same structure, the first two bytes represent the number of tags in the directories and all tags in IFD is stored in the same way:</div>
<ul><br />
<li>Tag (Bytes 0 – 1)</li>
<li>Type (Bytes 2 – 3)</li>
<li>Count (Bytes 4 – 7)</li>
<li>Value Offset (bytes 7 – 11)</li>
</ul>
<br />
<div style="text-align: justify;">
If the value can be represented in 4 bytes or smaller, it will be saved in the offset space else the offset will point the data and the “x” IFD value will be used.</div>
<div style="text-align: justify;">
The pointer to the next IFD is represented at the final of the IFD block, before the IFD values block. If you get the numbers of tags (2 first bytes) and times 12 (where 12 is the size of an IFD tag) you will get the offset to the next IFD postion (represented by 4 bytes).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I have a particular interest in the thumbnail, so as you can see in the explanation above (Exif organization), I need to have an IFD0 and IFD1 to have a thumbnail. IFD1 have two special tags, that points me to the thumbnail. The first one is the <em>JPEGInterchangeFormat</em> (0×0201) and the other one is the <em>JPEGInterchangeFormatLength</em> (0×0202).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The <em>JPEGInterchangeFormat</em> value is an offset to the beginning of the thumbnail and the <em>JPEGInterchangeFormatLength</em> contains the thumbnail size. With this two informations we are able to get any thumbnail embeded in JPEG Exif files ;P</div>
Anonymoushttp://www.blogger.com/profile/08307551500325815491noreply@blogger.com0