Pengcheng Xu's Placehttps://jsteward.moe/2021-08-12T18:00:00+08:00Apple Silicon Use Report2021-08-12T18:00:00+08:002021-08-12T18:00:00+08:00Pengcheng Xutag:jsteward.moe,2021-08-12:/apple-silicon-use-report.html<p>It's been over half a year since I accepted the donation of a M1 MacBook Air from <a href="https://t.me/jiegec">@jiegec</a> and I've been procrastinating for the promised use report. I finally got some free time to sum up the overall experience and rough edges during daily use, so here we are! While …</p><p>It's been over half a year since I accepted the donation of a M1 MacBook Air from <a href="https://t.me/jiegec">@jiegec</a> and I've been procrastinating for the promised use report. I finally got some free time to sum up the overall experience and rough edges during daily use, so here we are! While this post mainly focuses on the experience of the Apple Silicon-equipped version of 2020 MacBook Air, some reviews will probably be more about macOS and Apple itself since this is the first Mac I've extensively used. Let's get started!</p>
<p><img alt="About this Mac" src="/images/m1-ueber-diesen-mac.png"></p>
<p><strong>Note</strong>: this article is written on the M1 MacBook Air using Vim inside iTerm.</p>
<p><strong>Disclaimer</strong>: I'm in no form affliated with Apple and this is article does not serve as a recommendation or advertisement. The descriptions merely reflects my personal experience and YOUR MILEAGE MAY VARY. Having said that, welcome to share your ideas in the comments below.</p>
<h2>Chassis & Build Quality</h2>
<p>I'm generally pleased with the design and build quality of the hardware. As there're plenty of hardware reviews out there, I list a few points in my experience that's worth noting:</p>
<ul>
<li>The body is <strong>sturdy</strong> and does not flex, even if held on the edge with one hand; no wobbling or creaky sounds (unlike my 4-year-old ThinkPad)</li>
<li>The machine is light, but not too light such that it would easily fall off places</li>
<li>Speaker is really good - <strong>loud but clear</strong>, great spatial audio effects</li>
<li>Great battery life! Sustained use under light loads -> 12+ hours of battery life</li>
<li>Good screen with satisfying peak brightness, color accuracy, and resolution - the aspect ratio (16:10) is especially great</li>
<li>Personally I do not hate the keyboard with <strong>shallow key travel</strong>, but it can <strong>get quite loud</strong> under heavy use (I daily-drive an HHKB, so hard key presses). The key caps are large and good</li>
<li>Port selection (two USB 4 Type-Cs) is a minus but not deal breaker; used to <strong>carrying a hub</strong> after half year's use</li>
<li>Good microphones for conferencing, but they <strong>do not work</strong> when the <strong>lid is closed</strong>; seems like a <a href="https://www.reddit.com/r/macbook/comments/fit8wb/is_it_possible_to_enable_the_microphone_when_the/">security feature</a></li>
<li>No cooling fans, but has similar thermals with previous Intel Airs (I've used one on intern) - a <strong>quiet</strong> experience, which is very satisfying for my sensitive hearing</li>
</ul>
<p>One single flaw that's infamous among MacBooks, especially those sold in China, is that the bundled charger does not have a ground pin and the chassis can get tickling or even electrically shock the user when charging. This is still the case with the M1 MacBook Air, so make sure you <a href="https://twitter.com/mariotaku/status/496733277274013696">protect your sensitive body parts with sufficient shielding</a>.</p>
<h2>Performance - Sustained Loads</h2>
<p>The overall performance of the system is very satisfying. This includes the performance of native applications as well as x86 applications simulated with Rosetta 2 under a wide range of workloads. I'll share some experiences from daily use that really impressed me. The sustained load tests are more about native applications and not Rosetta-enabled ones, as most development tools should have Apple Silicon native support after over half a year since release.</p>
<h3>LLVM</h3>
<p>The M1 chip has so much better single-core performance that it (Apple Silicon M1, 16GB RAM) has performance on par with a full-blown Intel server (Dual <a href="https://ark.intel.com/content/www/de/de/ark/products/123547/intel-xeon-silver-4110-processor-11m-cache-2-10-ghz.html">Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz</a>, 64GB RAM) for a quick LLVM compile test. The Mac uses internal SSD for storage while the Intel server uses a SATA SSD. The results are obtained using LLVM monorepo commit <code>b5f3a128bf8cae46ccf0616477a4775fd168fd7c</code> with the following CMake options:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>cmake<span class="w"> </span>../llvm<span class="w"> </span>-DCMAKE_BUILD_TYPE<span class="o">=</span>Release<span class="w"> </span>-DLLVM_TARGETS_TO_BUILD<span class="o">=</span><span class="s2">"AArch64;RISCV;X86"</span><span class="w"> </span>-DLLVM_ENABLE_PROJECTS<span class="o">=</span><span class="s2">"clang;compiler-rt;libcxx;libcxxabi;lld"</span><span class="w"> </span>-G<span class="w"> </span>Ninja
</code></pre></div>
<p>On the Apple Silicon Mac:</p>
<div class="highlight"><pre><span></span><code><span class="go">[6214/6214] Linking CXX executable bin/lld</span>
<span class="go">ninja 8523,18s user 283,68s system 747% cpu 19:38,01 total</span>
</code></pre></div>
<p>On the Intel server:</p>
<div class="highlight"><pre><span></span><code><span class="go">[5538/5538] Generating ../../bin/llvm-readelf</span>
<span class="go">ninja 34268.50s user 1546.18s system 3084% cpu 19:21.05 total</span>
</code></pre></div>
<p>The 4+4 core M1 chip is only ~1.46% slower than the 16C32T Intel server for roughly the same workload! It's also worth noting that the M1 chip started out very fast but throttled very hard once hitting the 85 degree Celcius temperature wall, when it was about half way (~3500) through the 6214 build targets.</p>
<h3>Scala</h3>
<p>This effect can be demonstrated better with a spikey and less scalable workload, such as Scala compiling. With the <a href="https://github.com/KireinaHoro/chipsalliance-playground">chipsalliance playground</a>, the performance superiority of the M1 chip is more pronounced:</p>
<p>On the Apple Silicon Mac:</p>
<div class="highlight"><pre><span></span><code><span class="go">[#2] [1132/1132] playground.compile</span>
<span class="go">mill -j 0 _.compile 2,93s user 4,40s system 2% cpu 4:11,39 total</span>
</code></pre></div>
<p>On the Intel server:</p>
<div class="highlight"><pre><span></span><code><span class="go">[#22] [1132/1132] playground.compile</span>
<span class="go">mill -j 0 _.compile 15.94s user 11.09s system 6% cpu 6:56.15 total</span>
</code></pre></div>
<p>With a better thermal design (rather than the fan-less design on the Air), even better performance can be expected.</p>
<h2>Application Experience</h2>
<p>Most desktop applications that should run on a Mac runs on the M1 Mac. Native applications usually works without issues. The translated applications are usually games and desktop applications with GUI that proprietary and less popular, but still most of them works just fine. As desktop applications do not generally have a raw performance measure available, it is the overall experience that matters.</p>
<h3>Productivity</h3>
<p>Generally speaking, very well! A few things are still translated, but they work just fine.</p>
<ul>
<li>Chrome and Safari are of course all native and well-optimized. macOS supports separate windows for Chrome apps, allowing them to be pinned onto Dock separately with logo, so Google {Calendar,Keep,...} works great and integrated with the system.</li>
<li>Microsoft Office has always been choppy on Macs, but they work fine thanks to the native builds and thus improved performance.</li>
<li>Acrobat is still translated and and can stutter when quickly scrolling over complicated PDFs, but the editting works just fine when you focus on a small portion of the document. The builtin Preview.app is very smooth with good gesture support and can handle most PDF reading.</li>
<li>Zotero is still translated, but as it's not heavy everything's just fine.</li>
<li>IM and video conferencing stuff works without issues.</li>
<li>Mail.app sometimes uses excessive CPU and is usually resolved through relaunching.</li>
</ul>
<h3>Development</h3>
<p>Good experience, but can get confusing in complicated cases due to Rosetta messing with the toolchain.</p>
<ul>
<li>Homebrew has native M1 support at the time of writing, so most tools are already rebuilt natively (you can still get a few x86 casks though, but they still work just fine).</li>
<li>Most IDEs (that I use) now natively support M1, to name a few the JetBrains series, Xcode (of course!), VSCode. My experience is now so much better as it can handle much larger projects previously not feasible to use in an IDE (e.g. LLVM in CLion, Chipyard in IntelliJ, etc.) due to index getting too large.</li>
<li>Toolchains can get a little bit tricky as it is possible to mix up x86 and arm64 libraries, leading to symbol errors during linking. This can only be fixed in a case-by-case manner and <em>may lead to significant frustration</em>.</li>
</ul>
<h3>Multimedia</h3>
<p>Everything works just as on an Intel Mac.</p>
<ul>
<li>Video playback is usually smooth with the well-known IINA player. I sometimes need BluRay menu support, so VLC is a backup that always works.</li>
<li>Safari's audio system is <strong>at most times broken</strong>. The audio output is usually stuttering whenever I want to use it, and can be resolved by rebooting once or twice, but this annoys me <em>a lot</em>. I generally avoid Safari for anything related to audio.</li>
<li>A DSP plugin called <a href="https://www.sonarworks.com/Reference">Reference</a> is a good example of how drivers can still work after macOS banned third-party kexts. The application is translated and still works just fine, which wouldn't be possible if it's still using kexts and not the usermode driver framework.</li>
</ul>
<h3>Steam and games</h3>
<p>The few games with macOS support works really good.</p>
<ul>
<li>The Steam client browses libraries and store pages smoothly even if under translation.</li>
<li>Games (the few ones that support macOS) usually run just fine without framerate issues. Only if more games supported macOS...</li>
<li>Steam's in-house streaming works great using Moonlight if the latency is low.</li>
</ul>
<h3>Creativity</h3>
<p>I'm not a creator, but using the machine for some light creation is just fine.</p>
<ul>
<li>Omnigraffle works just fine through Rosetta.</li>
<li>Native Premiere Pro (Beta) can easily handle simple projects but quickly hits the ceiling after adding a few transitions and translations.</li>
</ul>
<h2>Virtualization</h2>
<p>The M1 chip, as an ARMv8.4 processor, comes with virtualization support. This can be utilized by either <a href="https://gist.github.com/citruz/9896cd6fb63288ac95f81716756cb9aa">QEMU with <code>hvf</code> acceleration</a> or Parallels. Parallels recently gets updated to 17 and improved guest graphical performance such that it is possible to run Windows-only games inside a Windows 10/11 on ARM VM, such as <a href="https://worldofwarships.asia/de/">World of Warships</a>. Windows on ARM comes with its own x64-to-aarch64 translation, but as the VM can't use the <a href="https://www.infoq.com/news/2020/11/rosetta-2-translation/">builtin Intel memory order mode</a> available in M1 chip, this suffers a performance penalty. The machine can get pretty hot and degrade in battery life when using a virtual machine.</p>
<h2>System Scheduling & Swapping</h2>
<p>macOS on M1 feels really snappy and responsive even under heavy loads, and this is partly due to the <a href="https://arstechnica.com/gadgets/2021/05/apples-m1-is-a-fast-cpu-but-m1-macs-feel-even-faster-due-to-qos/">OS scheduler</a> that performs fine-grain QoS towards the big.LITTLE clusters, as well as <a href="https://medium.com/codex/the-apple-m1-ssd-swapgate-is-a-massive-overreaction-50002ee23d0">eager swapping</a>. Personally, after 9 months of daily driving, my SSD has not experienced the extreme levels of wearing like some early reports:</p>
<div class="highlight"><pre><span></span><code>Available Spare: 100%
Available Spare Threshold: 99%
Percentage Used: 0%
Data Units Read: 47,212,581 [24,1 TB]
Data Units Written: 26,712,479 [13,6 TB]
Host Read Commands: 647,232,854
Host Write Commands: 279,577,858
</code></pre></div>
<p>I believe that daily use, plus the 16GB RAM, results in lower memory pressure (my memory pressure percentage usually stays below 40%). After all, the M1 is still a mobile platform, and the user shouldn't be putting too much load on it at the risk of device longevity (even so if taking the relatively poor thermals into consideration).</p>
<h2>Miscellenous Issues</h2>
<p>There are a few misc issues that does not belong to any specific category of applications, but rather to the system itself. They mildly affect my daily workflow but are not deal-breakers. Some of these issues may not be M1-specific but common to all Macs.</p>
<ul>
<li>The M1 chip only supports 1 external screen (despite the two USB-C ports)! It is not possible to use two monitors <em>natively</em>, although DisplayLink is a software/hardware solution to add more screens with its downsides.<ul>
<li>Thus, the machine does not support DisplayPort MST (Multi-Stream Transport). Multiple monitors in a daisy-chain configuration will be mirrored.</li>
</ul>
</li>
<li>A 2K (2560x1440), 24inch external screen won't be recognized by macOS as a Retina screen, thus scaling is not possible. I had to increase the size of the content on that screen manually (e.g. zoom in for Chrome/VSCode, increase font size for IntelliJ).</li>
<li>HDR settings for external secondary screen are not kept after unplugging. Setting the external screen as the main display <em>will</em> make the HDR settings persist, though.</li>
<li>Paragon NTFS installs a native kext. While basic functionalities work, it can get a bit unstable under heavy loads and crash the whole system.</li>
</ul>
<h2>Closing Remarks</h2>
<p>I'm generally pleased with the M1 MacBook Air as a product: it feels finished (not much rough edges), efficient, and performant. There are definitely areas to improve, such as the need for a 32GB or 64GB model, a better-designed chassis for cooling, etc, but this has been a great start. Even though I don't use iDevices, I believe I'll probably buy a M2 or M3 device again. Even for people not purchasing Apple products at all, as competition lowers prices and promotes innovation, it's always a good thing for consumers that Apple challenges on Intel and AMD in the mobile computing market.</p>
<p>Finally, <strong>all the best for Apple Silicon!</strong></p>Separate whitespace problems from real change in git with vim2021-08-10T01:27:00+08:002021-08-10T01:27:00+08:00Pengcheng Xutag:jsteward.moe,2021-08-10:/vim-git-diff-apply.html<p>Whitespace changes can be annoying, especially when we have colleagues that do not care so much about them. If we care about whitespace correctness, we probably have whitespace indicators and trimmers configured in vim:</p>
<div class="highlight"><pre><span></span><code><span class="c">" Removes trailing spaces</span>
<span class="k">function</span><span class="p">!</span> TrimWhiteSpace<span class="p">()</span>
<span class="c"> " Only strip if the b:noStripWhitespace variable isn't set</span>
<span class="k">if …</span></code></pre></div><p>Whitespace changes can be annoying, especially when we have colleagues that do not care so much about them. If we care about whitespace correctness, we probably have whitespace indicators and trimmers configured in vim:</p>
<div class="highlight"><pre><span></span><code><span class="c">" Removes trailing spaces</span>
<span class="k">function</span><span class="p">!</span> TrimWhiteSpace<span class="p">()</span>
<span class="c"> " Only strip if the b:noStripWhitespace variable isn't set</span>
<span class="k">if</span> exists<span class="p">(</span><span class="s1">'b:noStripWhitespace'</span><span class="p">)</span>
<span class="k">return</span>
<span class="k">endif</span>
%s<span class="sr">/\s*$/</span>/
<span class="s1">''</span>
<span class="k">endfunction</span>
<span class="c">" Control characters expansion</span>
<span class="k">set</span> <span class="nb">list</span> <span class="nb">listchars</span><span class="p">=</span><span class="k">tab</span>:»<span class="p">-,</span>trail:.<span class="p">,</span><span class="nb">eol</span>:↲<span class="p">,</span>extends:»<span class="p">,</span>precedes:«<span class="p">,</span>nbsp:%
<span class="k">au</span> <span class="nb">FileType</span> <span class="nb">diff</span> <span class="k">let</span> <span class="k">b</span>:noStripWhitespace<span class="p">=</span><span class="m">1</span>
<span class="k">au</span> <span class="nb">FileWritePre</span> * <span class="k">call</span> TrimWhiteSpace<span class="p">()</span>
<span class="k">au</span> <span class="nb">FileAppendPre</span> * <span class="k">call</span> TrimWhiteSpace<span class="p">()</span>
<span class="k">au</span> <span class="nb">FilterWritePre</span> * <span class="k">call</span> TrimWhiteSpace<span class="p">()</span>
<span class="k">au</span> <span class="nb">BufWritePre</span> * <span class="k">call</span> TrimWhiteSpace<span class="p">()</span>
</code></pre></div>
<p>Then our fixer will fix the whitespace errors our colleagues introduced as we work. Great, but things break down quickly when we try to use the <code>--patch</code> option of <code>git-add(1)</code>, <code>git-checkout(1)</code>, or similar utilities to selectively and interactively manipulate our changes, either for staging changes for commit, or partially dropping changes: whitespace corrections spam the interactive sessions, making it extremely difficult to navigate through. Well, with a bit of experience in editting <code>diff</code>s, this situation can be neatly handled using vim's pipe read and write functionality.</p>
<p>Note: this post is based on these two StackOverflow posts: <a href="https://stackoverflow.com/questions/6571643/how-can-you-combine-git-add-patch-p-mode-with-diffs-ignore-all-space">this</a> and <a href="https://stackoverflow.com/questions/24442069/is-there-a-way-to-show-only-whitespace-differences-with-git-diff/63904526#63904526">this</a>.</p>
<h2>Reading in a clean diff without whitespace changes</h2>
<p>What we want to do here is to get the real changes free of whitespace corrections for editting. The basic Ex command is:</p>
<div class="highlight"><pre><span></span><code>:set ft=diff | r !git diff -U0 -w
</code></pre></div>
<p>Piece by piece:</p>
<ul>
<li><code>set ft=diff</code> sets the current file type to diff, triggering any <code>autocmd</code>s there may be in our local config (e.g. to disable the space trimmer). This also gets us syntax highlighting.</li>
<li><code>r</code> reads the output of <code>git-diff(1)</code> into the current (anonymous) buffer for editting.</li>
<li><code>-U0</code> makes git generate 0 lines of context for the resulting patch. This is important! See the explanation below.</li>
<li>finally, <code>-w</code> ignores all <em>whitespace changes</em> in our diff, leaving them for process at a later time.</li>
</ul>
<p>Refer to the manual page for detailed explanations of options. Note that the 0 line context option is important because as we're ignoring whitespace changes, the context may be out of sync with HEAD. By dropping the contexts we make the patch applicable.</p>
<p>We can now proceed to edit the patches as normal, just like what we'd do using the <code>e</code> action during <code>--patch</code>.</p>
<h2>To stage a subset of changes (add to cache)</h2>
<p>We want to delete all currently unwanted changes in the patch. Use the following Ex command to stage the changes:</p>
<div class="highlight"><pre><span></span><code>:w !git apply --unidiff-zero --ignore-whitespace --cached
</code></pre></div>
<p>Piece by piece:</p>
<ul>
<li><code>w</code> writes the buffer to the input of <code>git-apply(1)</code>.</li>
<li><code>--unidiff-zero</code> makes git accept the <code>-U0</code> patch generated earlier.</li>
<li><code>--ignore-whitespace</code> makes git ignore whitespaces in context lines.</li>
<li><code>--cached</code> makes git apply into the staging area, keeping the tree untouched. This is essentially the <code>git-add(1)</code> behavior.</li>
</ul>
<h2>To drop changes from working tree</h2>
<p>We want to delete everything other than the changes we want to drop. The resulting patch is then applied <strong>in reverse</strong> with the following:</p>
<div class="highlight"><pre><span></span><code>:w !git apply -R --unidiff-zero --ignore-whitespace
</code></pre></div>
<p>Piece by piece:</p>
<ul>
<li><code>-R</code> applies the patch in reverse, essentially reverting the changes.</li>
<li>no <code>--cached</code> option means that we operate on the working tree.</li>
</ul>
<p>It is recommended to stash the working tree beforehand to avoid any unexpected loss of work.</p>Working with IP Packager and Integrator in Vivado using Tcl2021-02-20T13:00:00+08:002021-02-20T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2021-02-20:/vivado-ips-with-tcl.html<h2>Preface</h2>
<p>Vivado's IP Integrator offers convenient access to its vast collection of production-grade IPs for the designer to incorporate into their designs. Furthermore, they allow the designer to extend the experience to custom IP cores via the IP Packager, fostering better integration and promoting reuse within the Xilinx FPGA ecosystem …</p><h2>Preface</h2>
<p>Vivado's IP Integrator offers convenient access to its vast collection of production-grade IPs for the designer to incorporate into their designs. Furthermore, they allow the designer to extend the experience to custom IP cores via the IP Packager, fostering better integration and promoting reuse within the Xilinx FPGA ecosystem. Both the IP Integrator and IP Packager offer a GUI interface for straightforward access, but that can become quite tedious when working with larger designs with multiple IPs and/or multiple interfaces. Fortunately, with the help of some short Tcl, both the IP packaging (in the IP Packager) and instantiation (in the IP Integrator, or more commonly known as a <em>Block Design</em>) can be automated.</p>
<h3>Quick note</h3>
<p>Most of the affairs discussed in this article are, unlike other flows in Vivado, such as the Project/Non-Project Batch Mode, not very well documented. The following materials provided most of the information:</p>
<ul>
<li>UG912: Vivado Design Suite Properties Reference Guide</li>
<li>UG835: Vivado Design Suite Tcl Command Reference Guide</li>
<li>output in the <strong>Tcl Console</strong> when performing GUI </li>
<li><code>help</code> command</li>
<li><code>report_property</code> command</li>
<li>the Xilinx forum</li>
</ul>
<h2>IP Packager: mapping ports in interfaces</h2>
<p>The single most tedious process in packaging an IP is mapping the interfaces to top-level ports. While IP Packager offers interface inference, it bails out on the slightest discrepancies between top-level port names and the interface definitions (such as Vivado v. Chisel for AXI4). The interface definitions, as well as nearly all other functions in the IP Packager, can actually be manipulated with <code>ipx::*</code> commands; you can list them with <code>help ipx::*</code>.</p>
<p>Use <code>ips::add_bus_interface</code> to create an interface on the IP, set the abstraction type, bus type, and interface mode, and then create port maps with <code>ipx::add_port_map</code>. For example, if we're adding a BRAM interface called <code>$portName</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">set</span><span class="w"> </span>core<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>current_core<span class="k">]</span>
<span class="k">set</span><span class="w"> </span>intf<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>add_bus_interface<span class="w"> </span><span class="nv">$portName</span><span class="w"> </span><span class="nv">$core</span><span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span><span class="o">-</span>dict<span class="w"> </span><span class="k">[</span><span class="nb">list</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="nv">abstraction_type_vlnv</span><span class="w"> </span>xilinx.com:interface:bram_rtl:1.0<span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="nv">bus_type_vlnv</span><span class="w"> </span>xilinx.com:interface:bram:1.0<span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="nv">interface_mode</span><span class="w"> </span>master<span class="k">]</span><span class="w"> </span><span class="nv">$intf</span>
<span class="c"># can be stored as a dict</span>
<span class="nv">set_property</span><span class="w"> </span>physical_name<span class="w"> </span><span class="nv">${portName}</span>_address<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>add_port_map<span class="w"> </span>ADDR<span class="w"> </span><span class="nv">$intf</span><span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span>physical_name<span class="w"> </span><span class="nv">${portName}</span>_din<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>add_port_map<span class="w"> </span>DOUT<span class="w"> </span><span class="nv">$intf</span><span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span>physical_name<span class="w"> </span><span class="nv">${portName}</span>_dout<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>add_port_map<span class="w"> </span>DIN<span class="w"> </span><span class="nv">$intf</span><span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span>physical_name<span class="w"> </span><span class="nv">${portName}</span>_ce<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>add_port_map<span class="w"> </span>EN<span class="w"> </span><span class="nv">$intf</span><span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span>physical_name<span class="w"> </span><span class="nv">${portName}</span>_we<span class="w"> </span><span class="k">[</span><span class="nv">ipx</span><span class="o">::</span>add_port_map<span class="w"> </span>WE<span class="w"> </span><span class="nv">$intf</span><span class="k">]</span>
</code></pre></div>
<p>The above snippet is basically copied from the Tcl window after manually mapping one interface by hand. The defining keys for a BRAM interface are the abstract and bus VLNVs, which can be found in the command outputs. By storing the port map and bus interface objects, we can save time from the excessive <code>ipx::get_port_maps</code> and <code>ipx::get_bus_interfaces</code> commands. After testing the snippet out, it can then be incorporated into a <code>proc</code> to be used in further automation.</p>
<h2>IP Integrator: building a block design</h2>
<p>The block design can be built with Tcl calls rather than adding blocks and dragging connections between them, which quickly gets pretty tedious when more blocks get involved. You can choose to build from scratch, or base on the results from <strong>Export Block Design</strong>.</p>
<h3>Instantiating blocks and connecting nets</h3>
<p><code>create_bd_cell</code> is used to create about every type of BD cells, but in most cases we care about IPs the most. <code>set_property</code> is then used to customize the IP. For example, to create a Block Memory Generator that is a true dual port memory, as well as the corresponding controller:</p>
<div class="highlight"><pre><span></span><code><span class="k">set</span><span class="w"> </span>bmgName<span class="w"> </span>bmg_0
<span class="k">set</span><span class="w"> </span>bmg<span class="w"> </span><span class="k">[</span><span class="nv">create_bd_cell</span><span class="w"> </span><span class="o">-</span>type<span class="w"> </span>ip<span class="w"> </span><span class="o">-</span>vlnv<span class="w"> </span>xilinx.com:ip:blk_mem_gen:8.4<span class="w"> </span><span class="nv">$bmgName</span><span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span><span class="o">-</span>dict<span class="w"> </span><span class="k">[</span><span class="nb">list</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="nv">CONFIG.Memory_Type</span><span class="w"> </span><span class="k">{</span><span class="nv">True_Dual_Port_RAM</span><span class="k">}</span><span class="w"> </span><span class="err">\</span>
<span class="w"> </span><span class="nv">CONFIG.Assume_Synchronous_Clk</span><span class="w"> </span><span class="k">{</span><span class="nv">true</span><span class="k">}]</span><span class="w"> </span><span class="nv">$bmg</span>
<span class="k">set</span><span class="w"> </span>axiCtrlName<span class="w"> </span>axi_bram_ctrl_0
<span class="k">set</span><span class="w"> </span>axiCtrl<span class="w"> </span><span class="k">[</span><span class="nv">create_bd_cell</span><span class="w"> </span><span class="o">-</span>type<span class="w"> </span>ip<span class="w"> </span><span class="o">-</span>vlnv<span class="w"> </span>xilinx.com:ip:axi_bram_ctrl:4.1<span class="w"> </span><span class="nv">$axiCtrlName</span><span class="k">]</span>
</code></pre></div>
<p>Remember that the possible properties of a cell can be consulted via <code>report_property -all</code>. Make sure to check the reports and follow the RW/RO limitations, or the IP may misbehave.</p>
<p><code>connect_bd_net</code> is used to create simple nets (i.e. non-interfaces) while <code>connect_bd_intf_net</code> is for interfaces (e.g. AXI4, BRAM). <code>get_bd_{,intf_}{pins,ports}</code> can be used to find a port/pin by path. As we're using paths, saving the path name in a variable for later access will most likely be convenient. The following snippet assumes the pins <code>$clk</code> <code>$rstn</code> and <code>$axiM</code> hold the clock, active-low synchronous reset, and master AXI pins.</p>
<div class="highlight"><pre><span></span><code><span class="k">foreach</span><span class="w"> </span>a<span class="w"> </span><span class="k">{</span><span class="nv">A</span><span class="w"> </span>B<span class="k">}</span><span class="w"> </span><span class="k">{</span>
<span class="w"> </span><span class="nv">connect_bd_intf_net</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_intf_pins</span><span class="w"> </span><span class="nv">$axiCtrlName</span><span class="o">/</span>BRAM_PORT<span class="nv">$a</span><span class="k">]</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_intf_pins</span><span class="w"> </span><span class="nv">$bmgName</span><span class="o">/</span>BRAM_PORT<span class="nv">$a</span><span class="k">]</span>
<span class="k">}</span>
<span class="nv">connect_bd_intf_net</span><span class="w"> </span><span class="nv">$axiM</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_intf_pins</span><span class="w"> </span><span class="nv">$axiCtrlName</span><span class="o">/</span>S_AXI<span class="k">]</span>
<span class="nv">connect_bd_net</span><span class="w"> </span><span class="nv">$clk</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_pins</span><span class="w"> </span><span class="nv">$axiCtrlName</span><span class="o">/</span>s_axi_aclk<span class="k">]</span>
<span class="nv">connect_bd_net</span><span class="w"> </span><span class="nv">$rstn</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_pins</span><span class="w"> </span><span class="nv">$axiCtrlName</span><span class="o">/</span>s_axi_aresetn<span class="k">]</span>
</code></pre></div>
<h3>Mapping address segments</h3>
<p>There are two types of address segments in a block design for a specific IP such as the BRAM: </p>
<ul>
<li>A slave segment on an IP, which defines a mappable memory area</li>
<li>A segment in the master address space corresponding to a slave segment</li>
</ul>
<p><code>get_bd_addr_segs</code> uses paths to locate both kinds, so it's important to distinguish these two kinds correctly. <code>assign_bd_address</code> can be used to automatically assign (map) a slave segment into the master space. <code>set_property</code> can then be used on the newly-created segment to modify its <code>range</code> or <code>offset</code>. In the following snippet, we map the AXI BRAM controller into <code>0xdead0000</code> (stored in <code>$newOffset</code>). We assume that the master is called <code>$masterName</code> and the address space is <code>$addrSpaceName</code>.</p>
<div class="highlight"><pre><span></span><code><span class="c"># Slave Interface: S_AXI | Base Name: Mem0</span>
<span class="nv">assign_bd_address</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_addr_segs</span><span class="w"> </span><span class="nv">$axiCtrlName</span><span class="o">/</span>S_AXI<span class="o">/</span>Mem0<span class="k">]</span>
<span class="nv">set_property</span><span class="w"> </span>range<span class="w"> </span><span class="nv">$newOffset</span><span class="w"> </span><span class="k">[</span><span class="nv">get_bd_addr_segs</span><span class="w"> </span><span class="nv">$masterName</span><span class="o">/</span><span class="nv">$addrSpaceName</span><span class="o">/</span>SEG_<span class="nv">${axiCtrlName}</span>_Mem0<span class="k">]</span>
</code></pre></div>
<h3>Making hierarchies</h3>
<p>When the connections and address assignments are done, grouping automatically generated BD components into hierarchies is usually desirable. <code>group_bd_cells</code> is used for this purpose. <code>move_bd_cells</code> can be used to add a cell to an existing hierarchy.</p>
<div class="highlight"><pre><span></span><code><span class="k">set</span><span class="w"> </span>hierName<span class="w"> </span>hier_0
<span class="nv">group_bd_cells</span><span class="w"> </span><span class="nv">$hierName</span><span class="w"> </span><span class="nv">$bmg</span><span class="w"> </span><span class="nv">$axiCtrl</span>
</code></pre></div>
<p>Note that after creating a hierarchy, the paths for member cells will change, which may invalidate stored paths.</p>
<h3>Validate & save design</h3>
<p><em>Validate design</em> is an important step which allows IPs to propagate their parameters to connecting IPs. This mechanism supports various features such as width negotiation (bus or interface), link type negotiation (AXI4 or AXI4-Lite), and more complicated parameter propagation (such as BRAM operation mode). The following snippet validates, reorganizes (GUI-wise), and saves the current block design:</p>
<div class="highlight"><pre><span></span><code><span class="nv">validate_bd_design</span>
<span class="nv">regenerate_bd_layout</span>
<span class="nv">save_bd_design</span>
</code></pre></div>
<h3>Note on built-in BD automation</h3>
<p>While it is possible to use <code>apply_bd_automation</code> to simulate the automation process in block designs (e.g. instantiating BMGs or connecting AXIs), I recommend refraining from that and still connect things with explicit commands. This eliminates most surprises.</p>Gravity Access Wi-Fi on NUC VM2021-02-15T22:00:00+08:002021-02-15T22:00:00+08:00Pengcheng Xutag:jsteward.moe,2021-02-15:/nuc-gravity-wifi.html<h2>Preface</h2>
<p>Winter vacation means that most of my time is spent at home, instead of the dorm room. The consequence of this is that those dumb devices that require access to <em>Gravity</em> to properly function is now basically offline, as they're not capable of dialing in via IKEv2 or WireGuard …</p><h2>Preface</h2>
<p>Winter vacation means that most of my time is spent at home, instead of the dorm room. The consequence of this is that those dumb devices that require access to <em>Gravity</em> to properly function is now basically offline, as they're not capable of dialing in via IKEv2 or WireGuard for remote access by themselves. These are some examples of such devices (and where they're trying to communicate with):</p>
<ul>
<li>Kindle (<em>amazon.co.jp</em>)</li>
<li>Chromebooks (<em>google.com</em>)</li>
<li>Note that while Chromebooks, when set up, are capable of dialing a VPN, they are not in the <em>Welcome</em> screen</li>
<li>Android phones (<em>google.com</em>)</li>
<li>Same as above</li>
</ul>
<p>With the help of a NUC borrowed from <a href="https://t.me/Catofes">@Catofes</a>, we can set up a WiFi access point to <em>Gravity</em> for these devices. While it is true that the performance is going to be somewhat limited, we're not doing much other than syncing e-books, looking up Wikipedia definitions, and signing into Google accounts, so that should suffice. Here's a list of what we're going to set up:</p>
<ul>
<li><strong>Platform</strong>: VM, WiFi card passthrough, OS</li>
<li><strong>Backhaul</strong>: Gravity address allocation, WireGuard site-to-site</li>
<li><strong>Access</strong>: 802.11 - Hostapd, ISC DHCP - DHCP & DHCPv6, RADVD - SLAAC</li>
<li>Profit!</li>
</ul>
<p>Let's get started!</p>
<h2>The platform</h2>
<p>The NUC itself runs ESXi 7.0 to support some other tasks (mainly IoT stuff) that are not covered in this article. We're using a Debian 10 VM with the Intel 7265D WiFi card passthrough. A single core and 256 MB of RAM is enough to run the site-to-site VPN, hostapd, and all the address handout stuff.</p>
<p>A quick check with <code>iw list</code> shows that the 7265D is recognized and passthrough correctly, that it supports AP mode, but only in 2.4GHz:</p>
<div class="highlight"><pre><span></span><code>Wiphy phy0
...
Supported interface modes:
* IBSS
* managed
* AP
* AP/VLAN
* monitor
* P2P-client
* P2P-GO
* P2P-device
...
Band 1:
...
Frequencies:
* 2412 MHz [1] (22.0 dBm)
* 2417 MHz [2] (22.0 dBm)
* 2422 MHz [3] (22.0 dBm)
* 2427 MHz [4] (22.0 dBm)
* 2432 MHz [5] (22.0 dBm)
* 2437 MHz [6] (22.0 dBm)
* 2442 MHz [7] (22.0 dBm)
* 2447 MHz [8] (22.0 dBm)
* 2452 MHz [9] (22.0 dBm)
* 2457 MHz [10] (22.0 dBm)
* 2462 MHz [11] (22.0 dBm)
* 2467 MHz [12] (22.0 dBm) (no IR)
* 2472 MHz [13] (22.0 dBm) (no IR)
* 2484 MHz [14] (disabled)
...
Band 2:
...
Frequencies:
* 5180 MHz [36] (22.0 dBm) (no IR)
* 5200 MHz [40] (22.0 dBm) (no IR)
* 5220 MHz [44] (22.0 dBm) (no IR)
* 5240 MHz [48] (22.0 dBm) (no IR)
* 5260 MHz [52] (22.0 dBm) (no IR, radar detection)
* 5280 MHz [56] (22.0 dBm) (no IR, radar detection)
* 5300 MHz [60] (22.0 dBm) (no IR, radar detection)
* 5320 MHz [64] (22.0 dBm) (no IR, radar detection)
* 5500 MHz [100] (22.0 dBm) (no IR, radar detection)
* 5520 MHz [104] (22.0 dBm) (no IR, radar detection)
* 5540 MHz [108] (22.0 dBm) (no IR, radar detection)
* 5560 MHz [112] (22.0 dBm) (no IR, radar detection)
* 5580 MHz [116] (22.0 dBm) (no IR, radar detection)
* 5600 MHz [120] (22.0 dBm) (no IR, radar detection)
* 5620 MHz [124] (22.0 dBm) (no IR, radar detection)
* 5640 MHz [128] (22.0 dBm) (no IR, radar detection)
* 5660 MHz [132] (22.0 dBm) (no IR, radar detection)
* 5680 MHz [136] (22.0 dBm) (no IR, radar detection)
* 5700 MHz [140] (22.0 dBm) (no IR, radar detection)
* 5720 MHz [144] (22.0 dBm) (no IR, radar detection)
* 5745 MHz [149] (22.0 dBm) (no IR)
* 5765 MHz [153] (22.0 dBm) (no IR)
* 5785 MHz [157] (22.0 dBm) (no IR)
* 5805 MHz [161] (22.0 dBm) (no IR)
* 5825 MHz [165] (22.0 dBm) (no IR)
...
</code></pre></div>
<p>Note how every frequency in the 5GHz band is marked with <strong>no IR</strong>, which means no <em>initiating radiation</em> <a href="https://www.spinics.net/lists/linux-wireless/msg124066.html">apparently</a>. While this is most likely a regulatory limitation, soft or hard, I don't feel bothered to bypass this. We're sticking with 2.4GHz for now.</p>
<h2>The backhaul</h2>
<p>Backhaul in the current situation means bridging the incoming devices into <em>Gravity</em>. While it is possible to run a full node on the NUC VM, I don't really feel the need to do so: we're behind multiple layers of NAT (3 layers if including the ISP's), no good IPv6, and not stable enough to relay traffic for the rest of the network. Instead, a site-to-site VPN through WireGuard to a full node will serve the purpose. Note that this is very similar to the <em>Gravity</em> AP deployed at my dorm, which uses GRE for its lightweight and sharing the campus network with the full node. WireGuard should help with traversing NATs this time.</p>
<p>Note that no address is assigned to the WireGuard interface on the NUC VM; the gateway address sits on the wlan adapter. WireGuard works just fine forwarding these packets with the <code>fwmark</code>-based default route handling (from <a href="https://manpages.debian.org/testing/wireguard-tools/wg-quick.8.en.html"><code>wg-quick</code></a>).</p>
<h3><em>Gravity</em> address allocation</h3>
<p>As we're planning full dual-stack access, the following two blocks are alloted to this new site:</p>
<ul>
<li>IPv4: <code>10.172.194.128/25</code> (private)</li>
<li>IPv6: <code><prefix redacted>:cc20::/60</code></li>
</ul>
<p>The first host address of each network is the gateway and sits on the new site. These should not collide with anything else currently...</p>
<h2>The AP</h2>
<h3>hostapd - L2</h3>
<p>Hostapd runs the AP in 802.11n mode. The configuration is pasted below for future reference.</p>
<div class="highlight"><pre><span></span><code># /etc/hostapd/hostapd.wls192.conf
interface=wls192
hw_mode=g
channel=10
ieee80211d=1
country_code=CN
ieee80211n=1
wmm_enabled=1
ssid=JSteward Access
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP
wpa_passphrase=<redacted>
</code></pre></div>
<p>We also need to edit <code>/etc/hostapd/ifupdown.sh</code> to activate the <code>run-parts</code> hook of <code>ifupdown</code> to start and stop <code>hostapd</code> automatically. Insert this at the beginning:</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-f<span class="w"> </span>/etc/hostapd/hostapd.<span class="si">${</span><span class="nv">IFACE</span><span class="si">}</span>.conf<span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">IF_HOSTAPD</span><span class="o">=</span><span class="s2">"/etc/hostapd/hostapd.</span><span class="si">${</span><span class="nv">IFACE</span><span class="si">}</span><span class="s2">.conf"</span>
<span class="k">fi</span>
</code></pre></div>
<h3>ifupdown - L3</h3>
<p>As stated in the previous section, the gateway address sits on the wlan interface (for ARP/RA). The configuration is pasted below for future reference.</p>
<div class="highlight"><pre><span></span><code># /etc/network/interfaces
# Ethernet/lo configuration elided
# wifi AP
auto wls192
iface wls192 inet static
pre-up systemctl start --no-block wg-quick@seki
address 10.172.194.129/25
post-down systemctl stop --no-block wg-quick@seki
iface wls192 inet6 static
dad-attempts 0
address <prefix redacted>:cc20::1/60
</code></pre></div>
<p>Note the <code>--no-block</code> option to <code>systemctl</code>; this is necessary, otherwise the attempt to <code>ifup -a</code> at boot time will timeout due to name resolution issues (the Ethernet connection is not up yet). Having it delay a little bit solves all the problems.</p>
<h3>ISC DHCP & RADVD - address distribution</h3>
<p><code>radvd</code> and <code>isc-dhcp-server</code> works together to automatically hand out addresses to clients dual-stack. DHCPv4 is relatively simple:</p>
<div class="highlight"><pre><span></span><code># /etc/dhcp/dhcpd.conf
option domain-name "jsteward.moe";
option domain-name-servers 8.8.8.8, 8.8.4.4;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
subnet 10.172.194.128 netmask 255.255.255.128 {
range 10.172.194.130 10.172.194.254;
option routers 10.172.194.129;
}
</code></pre></div>
<p>DHCPv6 is just as simple. Note that there's no <code>routers</code> option for DHCPv6 though; the gateway (usually as an LL address) needs to be handed out though router advertisement. For consistency, DNS name server and search domain will be handed out via router advertisement as well.</p>
<div class="highlight"><pre><span></span><code># /etc/dhcp/dhcpd6.conf
default-lease-time 2592000;
preferred-lifetime 604800;
option dhcp-renewal-time 3600;
option dhcp-rebinding-time 7200;
allow leasequery;
option dhcp6.info-refresh-time 21600;
subnet6 <prefix redacted>:cc20::/60 {
range6 <prefix redacted>:cc20:b::1 <prefix redacted>:cc20:b::ff:ffff;
}
</code></pre></div>
<p>RADVD here works in the <a href="https://docs.opnsense.org/manual/radvd.html">so-called</a> <em>Assisted</em> mode:</p>
<blockquote>
<p>Assisted: Stateful configuration, address configuration provided by DHCPv6, although advertised routes can also be used on Stateless Address Autoconfiguration setups (SLAAC).</p>
</blockquote>
<div class="highlight"><pre><span></span><code># /etc/radvd.conf
interface wls192 {
AdvSendAdvert on;
AdvManagedFlag on;
AdvOtherConfigFlag on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
prefix <prefix redacted>:cc20::/60 {
AdvOnLink on;
AdvAutonomous off;
};
prefix <prefix redacted>:cc2a::/64 {
AdvOnLink on;
AdvAutonomous on;
};
RDNSS 2001:4860:4860::8888 2001:4860:4860::8844 { };
DNSSL jsteward.moe { };
};
</code></pre></div>
<p>The SLAAC block (<code><prefix redacted>:cc2a::/64</code>) uses a different prefix for easy distinguishing. Note that for unknown reasons using just one single <code>AdvAutonomous</code> <code>/60</code> prefix for both SLAAC and DHCPv6 didn't work; please leave a comment below if you have any clues.</p>
<h2>Results</h2>
<p>The system can support about 50Mbps downlink and 20Mbps uplink with an 2020 Macbook Air, translating to about 15% utility (all kernel time) on a single core. Considering that we're running this on 2.4GHz band, the bottleneck is most likely due to 7265D's Tx power limits and/or bad channel selection. In addition, the coverage is pretty bad as well, with signals barely usable outside the bedroom (the NUC is placed on the desk with plenty of cords on top). But, just as described in the preface, this should be sufficient for most of the use cases. Happy hacking!</p>Netbooting OpenBSD on SPARC642020-09-06T20:35:00+08:002020-09-06T20:35:00+08:00Pengcheng Xutag:jsteward.moe,2020-09-06:/sparc-openbsd-netboot.html<h2>What happened</h2>
<p>The trusty SPARC machine I obtained from someone on the Gentoo mailing list quietly had its SAS 10k RPM HDD failed; the machine has been running Solaris 11.3 for over two years since I've blown up the <a href="https://jsteward.moe/dual-disk-lvm-with-loopback.html">Gentoo setup</a>. As the machine is co-located inside PKU and …</p><h2>What happened</h2>
<p>The trusty SPARC machine I obtained from someone on the Gentoo mailing list quietly had its SAS 10k RPM HDD failed; the machine has been running Solaris 11.3 for over two years since I've blown up the <a href="https://jsteward.moe/dual-disk-lvm-with-loopback.html">Gentoo setup</a>. As the machine is co-located inside PKU and I do not have access to the server room, which is located on campus, due to the pandemic, the only possible way to revive the machine for probably some good is via netboot. Fortunately, the SPARC OpenBoot PROM supports netbooting via RARP/TFTP (or DHCP) and I have other machines that can serve as a boot server in the same broadcast domain. Let's get started!</p>
<h2>The boot protocol</h2>
<p>The OpenBSD <a href="https://man.openbsd.org/diskless">netboot protocol <code>diskless(8)</code></a> explains the boot process quite well:</p>
<blockquote>
<p>When booting a system over the network, there are three phases of interaction between client and server:</p>
<ul>
<li>The PROM (or stage-1 bootstrap) loads a <strong>boot program</strong>.</li>
<li>The boot program loads a <strong>kernel</strong>.</li>
<li>The kernel does NFS mounts for root and swap.</li>
</ul>
</blockquote>
<p>The first and second phases, on SPARC, relies on the early environment provided by <a href="https://tldp.org/HOWTO/SPARC-HOWTO-14.html">OpenBoot</a> for network access. </p>
<h3>The first stage</h3>
<p>The boot program loaded in this phase is the OpenBSD BOOT. The first stage translate into the following concrete sequence:</p>
<ul>
<li>OpenBoot reads settings from NVRAM or console (by user input) and selects network as boot medium</li>
<li>OpenBoot sends <a href="https://en.wikipedia.org/wiki/Reverse_Address_Resolution_Protocol">RARP</a> requests as broadcast to obtain an IP address</li>
<li>Assuming the host responding to RARP request is the boot server, OpenBoot pulls the relevant boot file via <a href="https://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol">TFTP</a></li>
<li>OpenBoot executes the pulled boot file</li>
</ul>
<p>The overall process is standardized as <a href="https://tools.ietf.org/html/rfc906">RFC906</a>, which dates back to 1984. The limitations of this protocol is obvious: the TFTP server and RARP server are required to run on the same host (or some severe hackery would be needed). A newer protocol, <a href="https://tools.ietf.org/html/rfc951">BOOTP (RFC951)</a>, is recommended for newer setups; we cannot deploy that here as I do not control the DHCP servers which, obviously, are run by the campus network operators.</p>
<h3>The second stage</h3>
<p>The OpenBSD BOOT then uses Bootparams and NFS protocol to obtain the root filesystem information, loads kernel from root, and passes relevant information to the kernel. Both protocols are part of the RPC protocol specified in <a href="https://tools.ietf.org/html/rfc5531">RFC5331</a>. Specifically,</p>
<ul>
<li>OpenBSD BOOT reads root and swap settings via broadcasted Bootparams</li>
<li>OpenBSD BOOT loads <code>bsd</code> and <code>bsd.rd</code> from root via NFS</li>
<li>OpenBSD BOOT executes <code>bsd</code>, passing root and swap info</li>
</ul>
<p>The kernel will then mount root and swap as specified during the later boot process.</p>
<h2>Preparing the boot server</h2>
<p>The boot server mainly has two sets of responsibilities during the whole boot process:</p>
<ul>
<li>Respond to the broadcast and unicast requests of various protocols (RARP, TFTP, Bootparams, NFS)</li>
<li>Serve the root and swap filesystems for the booted OpenBSD system</li>
</ul>
<h3>Boot daemon configurations</h3>
<p>Configuration files for relevant daemons are posted for future reference.</p>
<p><code>/etc/ethers</code>:</p>
<div class="highlight"><pre><span></span><code><mac address> <hostname>
</code></pre></div>
<p><code>/etc/hosts</code>:</p>
<div class="highlight"><pre><span></span><code><ip address> <hostname>
</code></pre></div>
<p><code>/etc/bootparams</code>:</p>
<div class="highlight"><pre><span></span><code><hostname> root=<server>:<root> swap=<server>:<swap>
</code></pre></div>
<p><code>/etc/exports</code>:</p>
<div class="highlight"><pre><span></span><code><root> <ip address>(rw,sync,no_subtree_check,no_root_squash,crossmnt,insecure)
<swap> <ip address>(rw,sync,no_subtree_check,no_root_squash,crossmnt,insecure)
</code></pre></div>
<p><code>/etc/default/tftpd-hpa</code>:</p>
<div class="highlight"><pre><span></span><code># /etc/default/tftpd-hpa
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/tftpboot"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure"
</code></pre></div>
<p>The OpenBSD BOOT file should reside in <code>/tftpboot</code> and symlinked to the output of the following:</p>
<div class="highlight"><pre><span></span><code>$ printf "%.2X%.2X%.2X%.2X\n" <ip address, separated with spaces>
</code></pre></div>
<h3>OpenBSD root filesystem preparation</h3>
<p>Follow the section of <em>Upgrade without the install kernel</em> in the <a href="https://www.openbsd.org/faq/upgrade67.html">OpenBSD Upgrade Guide: 6.6 to 6.7</a> to set up a working root filesystem on the NFS host. Additional tips are listed below.</p>
<ul>
<li>The upgrade guide expects <code>bsdtar</code> for unpacking. Make sure to keep the appropriate owners and groups. Refer to <code>tar(1)</code> for details.</li>
<li>To populate <code>/etc</code>, as most of the configuration files are out of <code>base</code>, unpack <code>/var/sysmerge/etc.tgz</code>. Refer to <code>sysmerge(8)</code> for details.</li>
<li>Run <code>/dev/MAKEDEV</code> to populate <code>/dev</code>. If setting up on a Linux host, which is most likely the case, the generated script will not run as-is; for example, Linux <code>mknod</code> does not accept multiple targets at once. Refer to <a href="https://ftp.openbsd.org/pub/OpenBSD/6.7/sparc64/INSTALL.sparc64">INSTALL.sparc64</a>, especially the <em>Net Boot or Diskless Setup Information</em> section, for detailed instructions. Rerun after system startup to fix permissions of the device nodes, which may be incorrect during creation on the Linux host.</li>
</ul>
<p>The system should be ready now.</p>Test for the new Disqus comment system2020-03-08T12:40:00+08:002020-03-08T12:40:00+08:00Pengcheng Xutag:jsteward.moe,2020-03-08:/disqus-comments-test.html<p>This site has been lacking a comment system for a long time (ever since its <a href="https://jsteward.moe/hello-world.html">establishment in 2016</a>). That was not a deliberate decision: I was simply too lazy to connect a proper comment system, and not much people wanted to leave a comment. I received almost no reach-outs due …</p><p>This site has been lacking a comment system for a long time (ever since its <a href="https://jsteward.moe/hello-world.html">establishment in 2016</a>). That was not a deliberate decision: I was simply too lazy to connect a proper comment system, and not much people wanted to leave a comment. I received almost no reach-outs due to the blog articles despite that the readers may easily find <a href="https://jsteward.moe/pages/about-and-contact.html">my contacts</a>.</p>
<p>Recently, however, in an effort to speed up the site build process on Travis, I purged all the themes that are not used on this site, and discovered that the theme actually had a Disqus template. Consequently, this should be the first article with a comment section the moment it's published.</p>
<blockquote>
<p>So what do you think? Leave a comment below!</p>
</blockquote>RISC-V Software Design: PS System - Edgeboard RISC-V Series2020-03-07T21:00:00+08:002020-03-08T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2020-03-07:/risc-v-software-design-part-b-edgeboard-series.html<h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>Besides <a href="https://jsteward.moe/risc-v-software-design-part-a-edgeboard-series.html">setting up RISC-V bootloader</a>, we need to have the PS side of Zynq set up properly for a proper system as well as ease …</p><h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>Besides <a href="https://jsteward.moe/risc-v-software-design-part-a-edgeboard-series.html">setting up RISC-V bootloader</a>, we need to have the PS side of Zynq set up properly for a proper system as well as ease of debugging. The BSP from ALINX will no longer be used - a new system will be used. The PS has the following roles in the system:</p>
<ul>
<li>Provide DDR and MMC to RISC-V</li>
<li>Provide reset signals</li>
<li>Provide Internet access</li>
<li>Control PWM fan on the board</li>
</ul>
<p>Note that the system block design has changed since <a href="https://jsteward.moe/risc-v-hardware-design-part-a-edgeboard-series.html">the previous introducing article</a>. The updated diagram and <a href="/images/updated-block-design.pdf">updated PDF</a> can be found below.</p>
<p><img alt="updated img" src="/images/updated-block-design.png"></p>
<p><em>Note that this article is not a detailed walkthrough of the entire process, and may subject to missing or inaccurate steps. You should only take this article as an overall guidance.</em></p>
<h2>PS bootloader & kernel setup</h2>
<p>The PS kernel will be built from the <a href="https://github.com/Xilinx/linux-xlnx/">linux-xlnx</a> tree to get support for all devices in the Zynq MPSoC chip. We will be using the <a href="https://www.xilinx.com/products/design-tools/embedded-software/petalinux-sdk.html">PetaLinux SDK</a> for generating the kernel, device tree, as well as FSBL, U-Boot and PMU firmwares. The PetaLinux SDK is also capable of creating a rootfs via the <a href="https://www.yoctoproject.org/">Yocto Project</a>, but we will only be using that feature very lightly.</p>
<p>Create the PetaLinux project and import hardware description exported from Vivado:</p>
<div class="highlight"><pre><span></span><code>$ petalinux-create --type project --template zynqMP --name rocket-zynqmp
$ cd rocket-zynqmp
$ petalinux-config --get-hw-description=..
</code></pre></div>
<p>Review the configuration, set hardware properties correctly, and save the configuration.</p>
<h3>Device tree generation</h3>
<p>My early attempts of modifying the decompiled device tree from ALINX BSP did not turn out very well: the compiled kernel often wouldn't boot with mysterious failures. The proper way to generate device tree for a ZynqMP platform is to utilize the device tree include mechanism: PetaLinux generates a basic configuration from exported hardware definition (<code>hwdef</code> file, which may contain a bitstream). A <code>system-user.dtsi</code> definition is then provided to make changes to the generated device tree.</p>
<div class="highlight"><pre><span></span><code>/include/ "system-conf.dtsi"
/ {
chosen {
bootargs = "earlycon console=ttyPS0,115200 clk_ignore_unused cpuidle.off=1";
};
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
riscv@40000000 { /* RISC-V */
reg = <0x0 0x40000000 0x0 0x3ff00000>;
};
};
};
&sdhci1 {
disable-wp;
no-1-8-v;
};
&ttc0 {
#pwm-cells = <1>;
status = "okay";
xlnx,ttc-clk0-freq-hz = <100000000>; /* PS_LSBUS_CLK */
xlnx,ttc-clk1-freq-hz = <100000000>;
xlnx,ttc-clk2-freq-hz = <100000000>;
xlnx,ttc-clk0-clksrc = <0>; /* the clock is internal */
xlnx,ttc-clk1-clksrc = <0>;
xlnx,ttc-clk2-clksrc = <0>;
};
</code></pre></div>
<p>The most notable and important modification is the <code>reserved-memory</code> node, which prevents PS kernel from using <code>0x40000000-0x7ff00000</code>, as the range is already allocated to RISC-V. Failing to do so may result in PS kernel crash (silently at most times) due to its data in the zone being destroyed by RISC-V.</p>
<p>The SDHCI1 controller is modified to add quirks to disable voltage switching and write-protect detection.</p>
<p>The TTC cell is added for PWM control of the fan attached onboard. A MOS is connected to the B11 pin in PL, and then drives the fan:</p>
<p><img alt="fan pwm" src="/images/fan-pwm.png"></p>
<h3>U-Boot special configuration</h3>
<p>As the previous article on RISC-V software design states, the RISC-V part of the system will use a MMC controller, notably the <code>mmc1</code> controller for persistent storage. Due to the Linux driver of Zynq's Arasan SDHCI host controller requiring a working IRQ, only one processor (ARM or RISC-V) may use the controller at a time. This is achieved via some U-Boot magic:</p>
<div class="highlight"><pre><span></span><code>env set bootcmd 'fatload mmc 0 ${netstart} image.ub && fdt addr ${fdtaddr} && fdt move ${fdtaddr} 0x7000000 && fdt addr 0x7000000 && fdt set /amba/mmc@ff170000 status disabled && env set bootargs earlycon console=ttyPS0,115200 clk_ignore_unused cpuidle.off=1 root=/dev/mmcblk0p2 rw rootwait && bootm ${netstart} - 0x7000000'
</code></pre></div>
<p>The actual boot command is then:</p>
<div class="highlight"><pre><span></span><code>fatload mmc 0 ${netstart} image.ub
fdt addr ${fdtaddr}
fdt set /amba/mmc@ff170000 status disabled
env set bootargs earlycon console=ttyPS0,115200 clk_ignore_unused cpuidle.off=1 root=/dev/mmcblk0p2 rw rootwait
bootm ${netstart} - ${fdtaddr}
</code></pre></div>
<p>What happened is then easily understood: after loading the U-Boot image <code>image.ub</code>, the FDT is edited to have the <code>mmc1</code> or <code>/amba/mmc@ff170000</code> node status set to <code>disabled</code>. The PS kernel then will not initialize the device or register IRQ handler for it. If this is not done, RISC-V will most likely fail to get an IRQ when it is expecting one, while the PS is constantly receiving IRQs with no apparent command issued (from its point of view). An <code>altboot</code> command is also stored to run PS with the <code>mmc1</code> node in <code>status = "okay"</code> if the SD card needs to be accessed from PS.</p>
<p>Make sure to save the environment variables to FAT after setting the variables:</p>
<div class="highlight"><pre><span></span><code>ZynqMP> env save
</code></pre></div>
<h3>Kernel options</h3>
<p>Two features need to be added to the PS kernel: PPP (<code>CONFIG_PPP</code>) and NFS server (<code>CONFIG_NFSD</code>); they are used for Internet and filesystem access. Also, to avoid conflict with the TTC PWM kernel module, make sure to disable the Cadence TTC timer driver (<code>CONFIG_CADENCE_TTC_TIMER</code>). Configure the PS kernel with:</p>
<div class="highlight"><pre><span></span><code>$ petalinux-config -c kernel
</code></pre></div>
<h4>Kernel modules</h4>
<p>A <a href="https://github.com/XiphosSystemsCorp/cadence-ttc-pwm">custom kernel module</a> for the TTC counter to work as a PWM device is needed for fan control. With <a href="https://www.xilinx.com/support/answers/55997.html">this Xilinx AR</a>, we can have the module build along the kernel. We can then acquire the installed module directory (as in <code>/lib/modules</code>) from the generated rootfs image:</p>
<div class="highlight"><pre><span></span><code>$ cd images/linux
$ mkdir rootfs-mount
$ sudo mount -o loop rootfs.ext4 rootfs-mount
$ cd rootfs-mount/lib/modules # tar the modules up and save them for later use
</code></pre></div>
<h3>BOOT.BIN</h3>
<p>After building the PetaLinux project via <code>petalinux-build</code>, the FSBL, PMU FW, bitstream, and some misc initialization files need to be packaged into a single <code>BOOT.BIN</code> file:</p>
<div class="highlight"><pre><span></span><code>$ petalinux-package --boot --fsbl zynqmp_fsbl.elf --fpga system.bit --u-boot --force
</code></pre></div>
<p>When there's a new bitstream available, but no big changes that would affect FSBL and U-Boot, we can simply regenerate the BIN file by swapping out <code>system.bit</code> in the above command.</p>
<h2>Rootfs setup</h2>
<p>For ease of accessing the RISC-V filesystem, I decided to put the RISC-V boot and root partition on the SD card (removable), while assigning PS with the eMMC storage. This resulted in a small problem: we need a system up and running to unpack the real rootfs into the eMMC chip. A tiny <code>buildroot</code> initramfs is used to perform this task (I failed to get yocto behave the way I wanted). The following tools are needed in the initramfs to perform this task:</p>
<ul>
<li>fdisk</li>
<li>dosfstools</li>
<li>e2fsprogs</li>
<li>tar</li>
<li>gzip</li>
</ul>
<p>Note that the busybox version of the tools above may show limited features. For example, you may need to untar a <code>.tar.gz</code> archive as follows:</p>
<div class="highlight"><pre><span></span><code>$ gzip -d < archive.tar.gz | tar xvf -
</code></pre></div>
<p>The real rootfs would be a <a href="https://archlinuxarm.org/platforms/armv8/generic">generic aarch64 Archlinux ARM</a> snapshot. As we have network access (onboard Ethernet) and a rather decent CPU (4-core Cortex-A53), Archlinux ARM would provide extreme flexibility over the fixed rootfs solution. Create a FAT32 partition followed by an EXT4 partition, copy <code>BOOT.BIN</code> and <code>image.ub</code> to the FAT32 partition, and untar the rootfs image into the EXT4 partition. Remember to untar the kernel modules acquired from the previous step as well.</p>
<h2>PL reset & clock</h2>
<p>The ZynqMP PS IP block provides up to 4 <code>pl_resetnx</code> ports for use to generate reset signals for PL logic besides the Power-On Reset mechanism. <a href="https://www.xilinx.com/support/answers/68962.html">This AR</a> clearly states that the reset port from PS IP block is simply a GPIO pin, and can be controlled from PS. A small script would be sufficient to control the pin status:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env bash</span>
<span class="nb">set</span><span class="w"> </span>-e
die<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$1</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="o">}</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$#</span><span class="w"> </span>-gt<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>die<span class="w"> </span><span class="s2">"usage: </span><span class="nv">$0</span><span class="s2"> [pin value]"</span>
<span class="k">fi</span>
<span class="nv">pin_number</span><span class="o">=</span><span class="m">510</span>
<span class="nv">gpio_dir</span><span class="o">=</span>/sys/class/gpio/gpio<span class="si">${</span><span class="nv">pin_number</span><span class="si">}</span>
<span class="nv">gpio_export</span><span class="o">=</span>/sys/class/gpio/export
toggle<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">gpio_dir</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="si">${</span><span class="nv">pin_number</span><span class="si">}</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">gpio_export</span><span class="si">}</span><span class="w"> </span><span class="c1"># enable pin 510 - PL Reset 1</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span>Pin<span class="w"> </span>direction:<span class="se">\ </span>
<span class="w"> </span>cat<span class="w"> </span><span class="si">${</span><span class="nv">gpio_dir</span><span class="si">}</span>/direction
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span>Old<span class="w"> </span>value:<span class="se">\ </span>
<span class="w"> </span>cat<span class="w"> </span><span class="si">${</span><span class="nv">gpio_dir</span><span class="si">}</span>/value
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$1</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">gpio_dir</span><span class="si">}</span>/value
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span>New<span class="w"> </span>value:<span class="se">\ </span>
<span class="w"> </span>cat<span class="w"> </span><span class="si">${</span><span class="nv">gpio_dir</span><span class="si">}</span>/value
<span class="o">}</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$#</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>Resetting<span class="w"> </span>PL...
<span class="w"> </span>toggle<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>sleep<span class="w"> </span><span class="m">1</span>
<span class="w"> </span>toggle<span class="w"> </span><span class="m">0</span>
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>Setting<span class="w"> </span>reset<span class="w"> </span>to<span class="w"> </span><span class="nv">$1</span>...
<span class="w"> </span>toggle<span class="w"> </span><span class="nv">$1</span>
<span class="k">fi</span>
</code></pre></div>
<p>The default polarity of the pin is low-active, but we're actually using it as high-active. This satisfies our use just well: the RISC-V part stays in reset until we pull down the reset pin.</p>
<p>If you look closely into the block design diagram, you will find out that we do not use <code>pl_clockx</code> ports from the PS IP block: this is intended, as Linux's <a href="https://www.kernel.org/doc/Documentation/clk.txt">Common Clock Framework</a> may mess with the RISC-V clock. A fixed oscillator output is used instead.</p>
<h2>PS TTC for PWM</h2>
<p>The <a href="https://ip.cadence.com/uploads/486/cdn-dsd-sys-design-ip-for-ttc-pdf">Triple Time Counter</a> inside PS can be used as a PWM source with the <a href="https://github.com/XiphosSystemsCorp/cadence-ttc-pwm">kernel module</a>. A simple bash script can then be used to control the PWM duty ratio of the fan:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env bash</span>
<span class="nb">set</span><span class="w"> </span>-e
die<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$1</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="o">}</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$#</span><span class="w"> </span>-ne<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>die<span class="w"> </span><span class="s2">"usage: </span><span class="nv">$0</span><span class="s2"> <duty ratio>"</span>
<span class="k">fi</span>
<span class="nv">pwm_number</span><span class="o">=</span><span class="m">0</span>
<span class="nv">pwm_dir</span><span class="o">=</span>/sys/class/pwm/pwm<span class="si">${</span><span class="nv">pwm_number</span><span class="si">}</span>
<span class="nv">pwm_export</span><span class="o">=</span>/sys/class/pwm/pwmchip0/export
<span class="nv">pwm_period</span><span class="o">=</span><span class="m">1000</span>
<span class="nv">pwm_duty</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"(</span><span class="nv">$1</span><span class="s2"> * </span><span class="si">${</span><span class="nv">pwm_period</span><span class="si">}</span><span class="s2">) / 1"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>bc<span class="k">)</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">pwm_dir</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="si">${</span><span class="nv">pwm_number</span><span class="si">}</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">pwm_export</span><span class="si">}</span><span class="w"> </span><span class="c1"># enable pwm #0 - Fan @ B11</span>
<span class="k">fi</span>
<span class="nb">echo</span><span class="w"> </span><span class="si">${</span><span class="nv">pwm_period</span><span class="si">}</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">pwm_dir</span><span class="si">}</span>/period
<span class="nb">echo</span><span class="w"> </span><span class="si">${</span><span class="nv">pwm_duty</span><span class="si">}</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">pwm_dir</span><span class="si">}</span>/duty_cycle
<span class="nb">echo</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">pwm_dir</span><span class="si">}</span>/enable
<span class="nb">echo</span><span class="w"> </span>Done.
</code></pre></div>
<p>Note that the PWM duty ratio does not reflect real RPM for a DC fan. For the Edgeboard FZ3, a mere 3% can keep the chip cool at 40 Celcius, compared to just 35 Celcius with 100%. To prevent damage in the case of forgetting to set PWM resulting in the fan not turning (without setup TTC will output constant 0), call the script in <code>rc.local</code> or some other mechanism.</p>RISC-V Software Design: Bootloader - Edgeboard RISC-V Series2020-03-07T11:00:00+08:002020-03-07T17:30:00+08:00Pengcheng Xutag:jsteward.moe,2020-03-07:/risc-v-software-design-part-a-edgeboard-series.html<h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>Following the complete of <a href="https://jsteward.moe/risc-v-hardware-design-part-b-edgeboard-series.html">setting up debug access</a>, we can finally start the configuration for the first piece of software that will run on the …</p><h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>Following the complete of <a href="https://jsteward.moe/risc-v-hardware-design-part-b-edgeboard-series.html">setting up debug access</a>, we can finally start the configuration for the first piece of software that will run on the RISC-V platform. The bootloader in this build consists of three parts: BootROM, OpenSBI, and U-Boot. Linux kernel will be the payload. </p>
<p><em>Note that this article will not cover all of the trial-and-errors appeared during the development process; only the important parts will be explained.</em></p>
<h2>The BootROM (first stage)</h2>
<p>Per <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/83c4a3104a0dfad18f6b69de625f97138fec8301/src/main/scala/Configs.scala#L50">this config line</a>, the reset vector of the generated RISC-V processor will be <code>0x10000</code>, which is located at the beginning of the BootROM which Rocket Chip generates. The <a href="https://github.com/chipsalliance/rocket-chip/tree/319b6c44450ccde38f33cd8a38dd80071a0b6528/bootrom">original BootROM from Rocket Chip</a> is simply a bootloop application: a <code>wfi</code> loop, which is anything but useful. A <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/83c4a3104a0dfad18f6b69de625f97138fec8301/bootrom/bootrom.S#L6">real BootROM</a> with some code borrowed from <a href="https://github.com/cnrv/fpga-rocket-chip">the CNRV repository</a>, is created, and does the following:</p>
<ul>
<li>Enable interrupt and probe the count of harts present on the system</li>
<li>Set up BootROM stack for each hart<ul>
<li>Hart 0 also sets up a mark for synchronization</li>
</ul>
</li>
<li>Jump to <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/83c4a3104a0dfad18f6b69de625f97138fec8301/bootrom/bootloader.c#L22">C code</a></li>
<li>Harts other than 0 wait until hart 0 signals initialization done<ul>
<li>Hart 0 continues initialization work</li>
</ul>
</li>
<li>Initialize UART</li>
<li>Set up machine trap (for debugging)</li>
<li>Load OpenSBI ELF from BRAM to DRAM</li>
<li>Signal the rest harts, branch to start of OpenSBI</li>
</ul>
<p>The OpenSBI ELF, with device tree (FDT) and U-Boot embedded inside as its payload, is stored in a piece of block RAM outside the Rocket Chip. The block RAM is generated via Xilinx's <a href="https://www.xilinx.com/products/intellectual-property/block_memory_generator.html">Block Memory Generator</a> IP. Compared to using Sifive's <code>TLROM</code>, Xilinx's BMG tends to correctly infer BRAM cells, while <code>TLROM</code> usually results in significant LUTRAM usage. The BMG IP accepts a <code>coe</code> file to initialize the memory cells and require re-synthesizing the IP to change the contents. The <code>coe</code> file is <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/83c4a3104a0dfad18f6b69de625f97138fec8301/bootrom/Makefile#L41">regenerated automatically</a> whenever the BootROM is recompiled.</p>
<h2>OpenSBI</h2>
<p>SBI serves as a bootloader and firmware for a RISC-V platform: it runs in M-mode, loads further stages of S-mode bootloaders (or the OS payload), hosts M-mode traps (timer, pseudo-instruction emulation, etc.), and provides service routines (<code>ecall</code> from S-mode) to operating systems. OpenSBI is the reference implementation of the SBI standard. A <a href="https://github.com/KireinaHoro/opensbi">fork of the original OpenSBI</a> repository contains the <a href="https://github.com/KireinaHoro/opensbi/tree/master/platform/edgeboard">platform definition for Edgeboard</a>, namely the initialization functions, device tree, and payload definition. The build routine is also <a href="https://github.com/KireinaHoro/opensbi/commit/f9749fee89201d65439f79d50a869bfb17428b2c">slightly modified</a> to accomodate U-Boot build routine in.</p>
<h3>Device tree</h3>
<p>The <a href="https://github.com/KireinaHoro/opensbi/blob/master/platform/edgeboard/edgeboard.dts">device tree</a> is based on the <a href="https://github.com/chipsalliance/rocket-chip/blob/8cec10850a217d49a34d24fc3ae799daed6bcf26/src/main/scala/diplomacy/DeviceTree.scala">DTS generated by Rocket Chip</a>, with the following additions to reflex peripherals outside the Rocket Chip SoC over AXI. The DTB, embedded inside OpenSBI, will have its start address passed to the next stage (U-Boot) per <a href="https://www.sifive.com/blog/all-aboard-part-6-booting-a-risc-v-linux-kernel">platform specification</a>.</p>
<h4>UART</h4>
<p>Two <a href="https://www.xilinx.com/support/documentation/ip_documentation/axi_uart16550/v2_0/pg143-axi-uart16550.pdf">AXI 16550</a> are present: <code>e0000000</code> (<code>ttyS0</code>) is used for RISC-V console (stdin/stdout/stderr), while <code>e1000000</code> (<code>ttyS1</code>) is connected to UART1 in Zynq's processing system for a PPP connection (will be explained in a follow-up article).</p>
<div class="highlight"><pre><span></span><code>axi_uart0: serial@e0000000 {
clock-frequency = <100000000>;
compatible = "ns16550a";
current-speed = <115200>;
device_type = "serial";
interrupt-parent = <&plic>;
interrupts = <1>;
reg = <0xe0000000 0x10000>;
reg-offset = <0x1000>;
reg-shift = <2>;
};
</code></pre></div>
<p>Note that the <code>ns16550a</code> compatible string is used instead of <code>ns16550</code>, otherwise the FIFOs will not be enabled, resulting in input overruns. Credit for this goes to <a href="https://t.me/imi415">@imi415</a>.</p>
<h4>MMC SDHCI</h4>
<p>A SDHCI controller is borrowed from Zynq PS to gain access to the SD card from RISC-V: this is used for a persistent storage to hold Linux rootfs.</p>
<div class="highlight"><pre><span></span><code>clk200: clk200 {
compatible = "fixed-clock";
#clock-cells = <0>;
clock-frequency = <200000000>;
u-boot,dm-pre-reloc;
};
sdhci0: mmc@ff170000 {
u-boot,dm-pre-reloc;
compatible = "arasan,sdhci-8.9a";
reg = <0xff170000 0x1000>;
clocks = <&clk200 &clk200>;
clock-names = "clk_xin", "clk_ahb";
interrupt-parent = <&plic>;
interrupts = <4>;
no-1-8-v;
disable-wp;
};
</code></pre></div>
<p>The <code>no-1-8-v</code> and <code>disable-wp</code> properties are borrowed from the stock FDT in the Edgeboard BSP, to work around problematic voltage switching and lack of Write Protect detection for MicroSD cards. The <code>clk200</code> node reflects the fixed AHB clock inside PS.</p>
<h4>The "chosen" node</h4>
<p>The "chosen" node provides some default values for software, notably U-Boot and Linux. Note that <code>earlycon=sbi</code> signals the Linux kernel to use SBI calls for early stage printing before kernel gets a chance to initialize the serial console (they're not used forever as it is expensive to do SBI calls).</p>
<div class="highlight"><pre><span></span><code>chosen {
bootargs = "earlycon=sbi root=/dev/mmcblk0p2 rootwait";
stdout-path = "/soc/serial@e0000000:115200";
};
</code></pre></div>
<h4><code>timebase-frequency</code> property</h4>
<p>The Linux kernel <a href="https://github.com/torvalds/linux/blob/63849c8f410717eb2e6662f3953ff674727303e7/arch/riscv/kernel/time.c#L21">expects</a> the <code>timebase-frequency</code> property to be under the <code>/cpus</code> node, but Rocket Chip somehow generates the property under each CPU core (e.g. <code>/cpus/cpu@0</code>), and this mismatch will result in an early panic. The property is moved to the correct location.</p>
<h3>Physical Memory Protection (PMP)</h3>
<p>The default <a href="https://content.riscv.org/wp-content/uploads/2017/05/riscv-privileged-v1.10.pdf">PMP</a> initialization of current OpenSBI is not suitable for our setup: the whole firmware will be masked totally inaccessible from S-mode and U-mode, but we have our device tree embedded inside OpenSBI. <a href="https://github.com/KireinaHoro/opensbi/commit/dc68ecd5c2b82bd431bb138412d5607acf0cd98b">A patch</a> lifts the limitation to read-only firmware; in this way, the DTB will be readable by the following U-Boot payload, and will then be relocated to somewhere else in DRAM.</p>
<h2>U-Boot</h2>
<p>A <a href="https://github.com/KireinaHoro/u-boot/tree/e1b6a8fa1e65915eedb0f31cf88cbefdf7fa45e7">U-Boot fork</a> has been created containing the board definition of the platform, along with a minimal <code>defconfig</code>. A <a href="https://github.com/KireinaHoro/u-boot/commit/e1b6a8fa1e65915eedb0f31cf88cbefdf7fa45e7#diff-bfdb6fca569d48da8315b5565b4d99d7R17">special environment variable</a> needs to be set prior to booting a Linux kernel to prevent erroneous device tree relocation. Credit for this goes to <a href="https://t.me/Icenowy">@Icenowy</a>.</p>
<p>After putting things together, as well as correctly setting up PS (described in a follow-up article), the BootROM prompt, OpenSBI information, and U-Boot prompt shall appear in order on the serial console.</p>RISC-V Hardware Design: Debug via BSCAN Chain - Edgeboard RISC-V Series2020-02-27T13:00:00+08:002020-02-27T15:40:00+08:00Pengcheng Xutag:jsteward.moe,2020-02-27:/risc-v-hardware-design-part-b-edgeboard-series.html<h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>After <a href="https://jsteward.moe/risc-v-hardware-design-part-a-edgeboard-series.html">setting up the hardware system</a>, we need a way to test if the system is actually running. The <a href="https://github.com/KireinaHoro/rocket-zynqmp/tree/master/bootrom">BootROM</a> is the location where the …</p><h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>After <a href="https://jsteward.moe/risc-v-hardware-design-part-a-edgeboard-series.html">setting up the hardware system</a>, we need a way to test if the system is actually running. The <a href="https://github.com/KireinaHoro/rocket-zynqmp/tree/master/bootrom">BootROM</a> is the location where the <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/master/src/main/scala/Configs.scala#L50">reset vector</a> points to, but to change the code in the BootROM one needs to resynthesize and reimplement the design, which can take up to an hour for a Zynq UltraScale+ device on the <a href="https://jsteward.moe/dev-system-setup-edgeboard-series.html">development system</a>. As a result, a fast, external debug method is needed, and GDB with <a href="https://github.com/riscv/riscv-openocd">OpenOCD</a> satisfies this requirement perfectly. This article covers how the BSCAN tunnel mechanism was adapted to work on Zynq UltraScale+ devices.</p>
<h2>BSCANE2 primitive on Xilinx devices</h2>
<p>While it is possible to use PL GPIO and connect a separate JTAG cable to directly access the Rocket Chip JTAG TAP, but I failed to plan ahead and <a href="https://jsteward.moe/zu3eg-purchase-edgeboard-series.html">had just purchased one JTAG cable</a>, and it's too much labour to switch between the cabling of Xilinx JTAG and that of the GPIO JTAG every time I want to program a bitstream or debug RISC-V. Fortunately, we can <a href="https://www.xilinx.com/support/documentation/sw_manuals/xilinx2018_1/ug974-vivado-ultrascale-libraries.pdf">find that</a> 7 Series and UltraScale architectures provide a <a href="https://forums.xilinx.com/t5/Other-FPGA-Architecture/BSCANE2-documentation/td-p/354185">BSCANE2</a> primitive:</p>
<blockquote>
<p>This design element allows access to and from internal logic by the JTAG Boundary Scan logic controller. This allows for communication between the internal running design and the dedicated JTAG pins of the device. Each instance of this design element will handle one JTAG USER instruction (USER1 through USER4) as set with the JTAG_CHAIN attribute.
To handle all four USER instructions, instantiate four of these elements and set the JTAG_CHAIN
attribute appropriately.
For specific information on boundary scan for an architecture, see the Configuration User
Guide for the specific device.</p>
</blockquote>
<p>This indicates that we can send JTAG commands to a PL design through the user registers accessible on the PL TAP from the dedicated JTAG port of the ZynqMP chip. The user chains, specifically <code>USER1</code> by default, powers the on-board debug facilities such as ILA, JTAG Master, or VIO:</p>
<p><img alt="debug hub chain" src="/images/debug-hub-user-chain.png"></p>
<p>Referring to <a href="https://www.xilinx.com/support/documentation/user_guides/ug470_7Series_Config.pdf">UG470 7 Series FPGAs Configuration User Guide</a> we can find out details about advanced features of JTAG on these devices besides user chains, for example <a href="https://jiege.ch/hardware/2020/02/09/program-artix7-on-macos/">programming a bitstream with OpenOCD from macOS</a>.</p>
<p>There has already been <a href="https://jiege.ch/hardware/2020/02/09/rocket-chip-bscan-analysis/">a mechanism</a> that utilizes the <code>USER4</code> chain to tunnel requests to the RISC-V TAP: <a href="https://github.com/sequencer/rocket-playground/blob/master/playground/src/fpga/FPGA.scala#L39">the BSCAN tunnel translator in Chisel</a> and <a href="https://github.com/riscv/riscv-openocd/blob/7cb8843794a258380b7c37509e5c693977675b2a/src/target/riscv/riscv.c#L361">the BSCAN tunnel implementation in RISC-V OpenOCD</a>. The solution will not work directly here however:</p>
<ul>
<li>The Zynq UltraScale+ platform uses a different JTAG IR for <code>USERx</code> chains:<ul>
<li><code>0x20</code> - <code>0x23</code> for 7-series FPGA</li>
<li><code>0x920</code> - <code>0x923</code> for Zynq UltraScale+</li>
<li>This can be confirmed in the <a href="https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/device-models/bsdl-models/zynq-ultrascale-plus-mpsoc.html">BSDL Models</a> released by Xilinx</li>
<li>Addressed in OpenOCD via <a href="https://github.com/KireinaHoro/riscv-openocd/commit/cb7f6be16f56a62fc1bdafe0030862e42446e6b2">commit</a></li>
</ul>
</li>
<li>The Zynq UltraScale+ TAP scan chain structure is different from normal FPGAs<ul>
<li>The user chains are located on the PL TAP (IR LEN=6) which is on the chain <strong>concatenated</strong> with the PS TAP (IR LEN=6), resulting in a single TAP with IR LEN=12</li>
<li>There is an ARM DAP sitting between the combined PS/PL TAP and TDO, which shifts the input and output data (DR contents) by one bit (bypass), addressed in the following commits:<ul>
<li>The BSCAN tunnel translator: <a href="https://github.com/KireinaHoro/rocket-zynqmp/commit/29b176484089042058a6d3fd6b22e63f9c8b32c8">commit</a>, <a href="https://github.com/KireinaHoro/rocket-zynqmp/commit/ce6a810722530894ff2f9f38efb6ed26386f1fb5">commit</a>, handles the <strong>input delay</strong> of one TCK cycle due to OpenOCD TAP selection</li>
<li>OpenOCD: <a href="https://github.com/KireinaHoro/riscv-openocd/commit/11238e1eb4ecb883ee36fcf24668187c782ade5a">commit</a>, handles the <strong>output delay</strong> of one TCK cycle due to the ARM DAP</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>A picture from the article <a href="http://nahitafu.cocolog-nifty.com/nahitafu/2018/10/zynq-ultrasca-1.html">「ZYNQ UltraScale+ MPSoCのJTAGのしくみ」</a>, which explains the JTAG structure of Zynq UltraScale+ in detail:</p>
<p><img alt="zujtag" src="/images/zujtag-nahitafu.png"></p>
<h2>GDB</h2>
<p>After understanding the user chain mechanisms for Zynq UltraScale+, we can finally attach GDB to our RISC-V core. An OpenOCD config originally for debugging the APUs (Cortex-A53) on ZynqMP was <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/master/openocd.cfg">adapted</a> to enable RISC-V debugging. Stop hardware server (to avoid <code>libusb</code> conflict with OpenOCD), launch OpenOCD, and attach gdb:</p>
<div class="highlight"><pre><span></span><code>(the hwserver)
# systemctl stop hwserver
# openocd -f openocd.cfg -c "init_riscv"
</code></pre></div>
<div class="highlight"><pre><span></span><code>(the development machine)
# riscv64-linux-gnu-gdb
...
(gdb) target remote 192.168.79.129:3333
</code></pre></div>
<p>The following picture shows loading an OpenSBI ELF (which will be covered in the follow-up software articles) to DDR at <code>0x40000000</code>:</p>
<p><img alt="gdb load" src="/images/gdb-load-opensbi.jpg"></p>RISC-V Hardware Design: System & Block Design - Edgeboard RISC-V Series2020-02-25T14:40:00+08:002020-02-27T15:40:00+08:00Pengcheng Xutag:jsteward.moe,2020-02-25:/risc-v-hardware-design-part-a-edgeboard-series.html<h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>After <a href="https://jsteward.moe/dev-system-setup-edgeboard-series.html">setting up the development environment</a> and <a href="https://jsteward.moe/zu3eg-purchase-edgeboard-series.html">testing the board</a>, we can finally start designing the RISC-V hardware system. This part of the series describes …</p><h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>After <a href="https://jsteward.moe/dev-system-setup-edgeboard-series.html">setting up the development environment</a> and <a href="https://jsteward.moe/zu3eg-purchase-edgeboard-series.html">testing the board</a>, we can finally start designing the RISC-V hardware system. This part of the series describes the configuration of the <a href="https://github.com/chipsalliance/rocket-chip">Rocket Chip SoC generator</a>, which is used to generate the processor IP used in this project, as well as the <em>"uncore"</em> peripheral setup. The hardware setup are published in the <a href="https://github.com/KireinaHoro/rocket-zynqmp">rocket-zynqmp</a> repository. A follow-up article, also about hardware platform design, will focus on enabling GDB debugging for the RISC-V core.</p>
<h2>Configuring Rocket Chip</h2>
<p>Rocket Chip is an SoC generator, which <a href="https://riscv.org/wp-content/uploads/2015/01/riscv-rocket-chip-generator-workshop-jan2015.pdf">is necessary</a> for instantiating the Rocket Chip CPU core and using it. The CPU core and uncore components are implemented with the <a href="https://www.chisel-lang.org/">Chisel/FIRRTL HDL</a>, while the SoC generator, <a href="https://github.com/chipsalliance/rocket-chip/tree/master/src/main/scala/diplomacy">Diplomacy</a>, which is used to automatically negotiate various bus structures between components in the SoC, is written in Scala. Thanks to the flexibility of Chisel generators as well as the Scala language, the Rocket Chip can be configured to adapt to different needs quite easily.</p>
<p>The exact config I have been using on the Edgeboard can be found <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/master/src/main/scala/Configs.scala#L10">here</a>. Listing the main differences from <code>DefaultConfig</code>:</p>
<ul>
<li>Main memory (DRAM) has been relocated to <code>0x40000000</code>, with a size of <code>0x3ff00000</code>.<ul>
<li>The Edgeboard has 2GB of DDR4 RAM available. Under this configuration, the APU in Zynq PS can use <code>0x00000000</code>, size <code>0x40000000</code> (1GB) as main memory, and the higher ~1GB can be used by RISC-V.<ul>
<li>The highest 16MB of RAM is reserved by the PMU in ZynqMP.</li>
</ul>
</li>
</ul>
</li>
<li>MMIO has been relocated to <code>0xe0000000</code>.<ul>
<li>According to <a href="https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf">UG1085</a>, most of the PS peripherals are mapped in <code>0xe0000000</code> to <code>0xffffffff</code>. In order to use PS peripherals in RISC-V, specifically GEM and MMC, we have to follow this arrangement as well.</li>
</ul>
</li>
<li>There are 4 interrupts to the interrupt controller, listed in the order of IRQ:<ul>
<li>UART</li>
<li>Ethernet (Zynq GEM)</li>
<li>Ethernet Wake</li>
<li>MMC</li>
</ul>
</li>
<li>The system runs at 100MHz.<ul>
<li>This is used for a correct <code>timebase-frequency</code> as well as other frequency specifications in the generated device tree.<ul>
<li><em>Although the generated device tree is <a href="https://github.com/KireinaHoro/opensbi/blob/64b6b1c96ad30910e60a219cf908455ceb017658/platform/edgeboard/edgeboard.dts">not used</a> after all...</em></li>
</ul>
</li>
</ul>
</li>
<li>8 hardware breakpoints, instead of the default of 2, are instantiated.<ul>
<li>2 of them are just not enough for any proper debugging with GDB later on.</li>
<li>You may have heard of <a href="http://www.nynaeve.net/?p=80">software breakpoints</a>, but they don't work well in the situation we're in. The follow-up article about debugging will cover this.</li>
</ul>
</li>
<li>1 big core (RV64GC) is instantiated.<ul>
<li>Unlike the <a href="https://www.xilinx.com/products/boards-and-kits/ek-u1-zcu102-g.html#overview">ZCU102 with ZU7EG</a> which can house 4 big cores with barely over 50% LUT use, the Edgeboard uses ZU3EG and will house 1 big core with about 55% LUT use.</li>
</ul>
</li>
<li>The reset vector has been changed to <code>0x10000</code>.<ul>
<li>The original setup has it at <code>0x10040</code>, which is just a bootloop application. This change enables meaningful code to be run on the processor.</li>
</ul>
</li>
</ul>
<p>Besides configuration, a <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/master/src/main/scala/Top.scala">top module implementation and optional wrapper</a> are needed for actual instantiation of the SoC. We will not go through how the various traits contribute to the final generation of the SoC, or how Diplomacy works. After running the generator and FIRRTL compiler, we get our generated Verilog file, which contains the <code>RocketTop</code> module ready for simulation or synthesis.</p>
<h2>Xilinx IP Block Design</h2>
<p>The Rocket Chip SoC instantiated in this project communicates with the <em>external world</em> over AXI4 MM interfaces, and is instantiated in Vivado in a Block Design or, officially, the <a href="https://reference.digilentinc.com/vivado/getting-started-with-ipi/start">Vivado IP Integrator</a>. This is the usual flow for a complex system design, especially one with some kind of CPU, on a Xilinx device with the Vivado suite. Despite that it's possible to eliminate MMIO entirely and use all peripherals from SiFive (as done by <a href="https://t.me/Sequencer">@Sequencer</a> <a href="https://github.com/sequencer/rocket-playground/blob/master/playground/src/fpga/FPGA.scala#L89">here</a>), we still adopt the IP Integrator flow.</p>
<p>To save time that would be wasted if every time Rocket Chip needed re-synthesizing despite only small changes to the BootROM, AXI topology, or <a href="https://www.xilinx.com/products/intellectual-property/axis_ila.html">debug cores</a>, we package the Rocket Chip module as a separate IP such that the synthesis results can be cached. Other IP modules used in the project are:</p>
<ul>
<li>Zynq UltraScale+ MPSoC (the PS block)<ul>
<li>DRAM access, peripheral access</li>
<li>System reset<ul>
<li>The PS fabric clock output was deliberately avoided due to its unpredictable behavior (which caused a lot of confusions together with <a href="https://t.me/jiegec">@jiegec</a>)</li>
</ul>
</li>
</ul>
</li>
<li>AXI UART16550<ul>
<li>For serial access<ul>
<li>The board lacked a proper board file, so it's required to only make UART <code>sin</code> and <code>sout</code> external, or unconstrainted ports that does not actually exist on board will prevent bitstream generation</li>
</ul>
</li>
</ul>
</li>
<li>AXI BRAM Controller and Block Memory Generator<ul>
<li>Used together for hosting the Secondary BootLoader - OpenSBI with U-Boot as payload. This will be explained in detail in the follow-up RISC-V software system articles.</li>
</ul>
</li>
<li>Various utility IPs<ul>
<li>AXI Interconnect for data width, protocol conversion and xbar feature</li>
<li>JTAG to AXI master for debugging memory and device access</li>
<li>VIO (Virtual Input/Output) for RISC-V reset signal (Edgeboard does not have any push buttons wired to PL)</li>
<li>Processor System Reset, Clocking Wizard for clock and reset generation</li>
<li>Utility Vector Logic, Concat, Constant for manipulating basic signals</li>
</ul>
</li>
</ul>
<p>The complete block design is as shown below. <a href="/images/block-design-rocket-chip.pdf">Download PDF</a> if you want to examine the details. Also check out the <a href="https://github.com/KireinaHoro/rocket-zynqmp/blob/master/vivado/src/design_1_bd.tcl">TCL script</a> for recreating the block design in Vivado.</p>
<p><img alt="block design" src="/images/block-design-rocket-chip.png"></p>
<p>With all the setup done, we can now launch the implementation run in Vivado, generate a bitstream, and get ready for on board tests with GDB through JTAG.</p>
<h2>Footnote: a PS DDR complication</h2>
<p>During the development process, I've spent a lot of time trying to debug weird problems with DDR access from RISC-V. Sometimes the DRAM appears to be working and will host C stacks for some time, but then the stack gets corrupted mysteriously. Loading code into DDR won't work at all: the content is different from what had been loaded when trying to read back.</p>
<p>The debug process went rather extensive, involving trying to memtest from PS and PL, with RISC-V or without RISC-V (with JTAG to AXI master), over different interconnect ports PS has to offer. Finally, in an orchestrated debug attempt involving both PL and PS, the following pattern was identified:</p>
<p><img alt="mem dump" src="/images/riscv-ddr-malfunction.jpg"></p>
<p>The pattern was that PS wrote continuous <code>Hello, world!</code> pattern to DRAM assigned for RISC-V, and RISC-V writing simple memtest incremental patterns word-by-word. Obviously this is a data width problem. After changing the AXI slave data width from 64 to 128, everything worked like a charm.</p>
<p><img alt="axi width" src="/images/ps-axi-data-width.png"></p>
<p>This behavior is counter-intuitive and nowhere to be seen in the <a href="https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf">Technical Reference Manual</a>. I would appreciate it greatly if someone familiar with the ZynqMP can explain to me why this happened. If you have any clues, please don't hesitate to contact me.</p>Meet the Baidu Edgeboard ZU3EG - Edgeboard RISC-V Series2020-02-25T14:40:00+08:002020-02-25T17:50:00+08:00Pengcheng Xutag:jsteward.moe,2020-02-25:/zu3eg-purchase-edgeboard-series.html<h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>Someone in the TUNA Embedded Telegram group mentioned that there has been a cheap Xilinx ZynqMP UltraScale+ board made by Baidu back in January, 2020 …</p><h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>Someone in the TUNA Embedded Telegram group mentioned that there has been a cheap Xilinx ZynqMP UltraScale+ board made by Baidu back in January, 2020. The board plus some necessary accessories (a pre-installed fan, a USB-A to mini USB-B cable, a power supply, a 16 GB SD card) costed <a href="https://item.taobao.com/item.htm?spm=a1z09.2.0.0.3c502e8d4sqiFv&id=608706046387&_u=p291v2au8805">1075 CNY</a> on the Chinese online shopping site Taobao at the time of purchase. With <a href="https://www.xilinx.com/support/documentation/data_sheets/ds894-zynq-ultrascale-plus-overview.pdf">Xilinx ZynqMP XA ZU3EG (Automotive grade)</a> it's quite a bargain. This article is to introduce the basic features and the hardware setup for the RISC-V project.</p>
<h2>Board features</h2>
<p><a href="https://ai.baidu.com/tech/hardware/deepkit">The board (Edgeboard FZ3)</a> features a handful of low-speed and high-speed I/O ports, along with quite a few peripherals available for use. Due to its designed usecase of performing FPGA-accelerated AI inference, not too much I/O is connected to the PL part of the Zynq MPSoC. Fortunately we still have some GPIO pins connected to the PL, and that should be sufficient as we only need a serial port from PL for RISC-V. The exact features used will be discussed in the <a href="https://jsteward.moe/risc-v-hardware-design-part-a-edgeboard-series.html">RISC-V Hardware Design: System & Block Design - Edgeboard RISC-V Series</a> article.</p>
<p>Gathering the various I/O and peripheral resources as follows (fabricated peripheral cores in the PS are not listed here):</p>
<ul>
<li>ZynqMP system (XAZU3EG-1SFVC784I)</li>
<li>PS side<ul>
<li>2GB DDR4 DRAM: Micron MT40A512M16LY-062EIT</li>
<li>256Mbit QSPI Flash: Micron MT25QU256ABA1EW9-0SIT</li>
<li>8GB eMMC Flash: Micron MTFC8GAKAJCN-4M</li>
<li>EEPROM: Microchip 24LC04B-I/SN</li>
<li>Mini DisplayPort with 2 GT lanes</li>
<li>USB: 1x 3.0, 1x 2.0</li>
<li>GPHY: Micrel KSZ9031RNXIC, with RJ45 port</li>
<li>USB UART: Silicon Labs CP2102, with miniUSB-B</li>
<li>MicroSD slot</li>
<li>PCIe 2.0 x1, with 1 GT lane</li>
<li>CAN: TI SN65HVD232, 2 pins in 44-pin header</li>
<li>JTAG (without PSRST)</li>
</ul>
</li>
<li>PL side<ul>
<li>RS485: MAX3485, 2 pins in 44-pin header</li>
<li>GPIO 38x (excluding VCC/GND, CAN, RS485) in 44-pin header</li>
<li>MIPI 25-pin</li>
<li>BT1120 32-pin</li>
<li>LED 1x</li>
<li>25MHz reference clock: Silicon Labs Si5332BD11025-GM2</li>
</ul>
</li>
</ul>
<h2>Additional accessories and modifications</h2>
<p>The board needs some additional accessories to work as the platform for the RISC-V project:</p>
<ul>
<li>The 44-pin GPIO port on the board didn't come with pin headers soldered. A 44-pin (in 2x22) header needs to be purchased and then soldered onto the board.</li>
<li>Consequently, a solder iron kit is also needed (the iron, solder, rosin flux, etc.)</li>
<li>The JTAG debug port, which is used for programming FPGA bitstreams, does not have a debug cable chip integrated. A Digilent SMT2 JTAG cable needs to be purchased.</li>
<li>The board's GPIO headers are of 2.00mm width. Some 2.00mm-to-2.54mm Dupont cable is needed.</li>
<li>Finally, the UART signal from RISC-V system requires a USB TTL to be read by PuTTY on the development system.</li>
</ul>
<p>The order information for the required parts can be found below.</p>
<p><img alt="edgeboard order" src="/images/edgeboard-order.png"></p>
<p><img alt="jtag cable order" src="/images/jtag-cable-order.png"></p>
<p><img alt="male header ttl order" src="/images/male-header-ttl-order.png"></p>
<p>It took about a week to collect all the parts in mail, compared to which would be about three days if it were not for the recent COVID-19 event. After soldering the GPIO headers and connecting the cables, the ZynqMP system is ready for test.</p>
<h2>Testing the system</h2>
<p>The system was configured to boot from SD card, which comes with a FSBL that holds Baidu's accelerator bitstream, and a root filesystem with self-hosting tools. Tools like GCC, Clang, and CMake are ready in the image, kudos for the engineers at Baidu.</p>
<p><img alt="ps boot" src="/images/zynqmp-ps-boot.png"></p>
<p>Note for PuTTY users: disabling flow control is necessary for proper console input.</p>
<p><img alt="flow control" src="/images/putty-flow-control.png"></p>
<p>After starting <code>hwserver</code> as described in <a href="https://jsteward.moe/dev-system-setup-edgeboard-series.html">the previous article</a> and connecting Vivado to the <code>hwserver</code> instance, we can see that the two TAPs are correctly recognized, and data from various on-chip sensors can be read from the two SysMon devices.</p>
<p><img alt="vivado hwserver" src="/images/vivado-hwserver.png"></p>
<p>Further functional tests about the shipped AI core can also be performed following the hardware manual from Baidu and ALINX (designer and manufacturer of PCB), and will not be covered here.</p>
<h2>Gallery</h2>
<p>Main visual of the Edgeboard in working state:</p>
<p><img alt="key visual" src="/images/zu3eg-main-view.jpg"></p>
<p>Unsoldered GPIO pads:</p>
<p><img alt="unsoldered" src="/images/zu3eg-unsoldered.jpg"></p>
<p>After soldering:</p>
<p><img alt="soldered" src="/images/zu3eg-solder-gpio.jpg"></p>
<p>Solder iron used:</p>
<p><img alt="iron" src="/images/cheap-solder-iron.jpg"></p>
<p>JTAG cable used:</p>
<p><img alt="jtag" src="/images/jtag-cable.jpg"></p>Development System Setup - Edgeboard RISC-V Series2020-02-25T12:30:00+08:002020-02-25T12:30:00+08:00Pengcheng Xutag:jsteward.moe,2020-02-25:/dev-system-setup-edgeboard-series.html<h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>A developer needs a system to work with. For embedded developers, a development system does not refer to the platform his work is going to …</p><h2>Intro</h2>
<p>This article is a part of the <a href="https://jsteward.moe/edgeboard-series.html">Edgeboard RISC-V series</a>. Check out other articles as well if you came in through a search engine.</p>
<p>A developer needs a system to work with. For embedded developers, a development system does not refer to the platform his work is going to run on (that's called the <em>target</em>), but the system he uses daily to produce the work. This article introduces the setup of my daily driver, a <a href="https://www.lenovo.com/us/en/laptops/thinkpad/thinkpad-t-series/ThinkPad-T470p/p/22TP2TT470P">Lenovo ThinkPad T470p</a> with <a href="https://ark.intel.com/content/www/us/en/ark/products/97496/intel-core-i7-7820hq-processor-8m-cache-up-to-3-90-ghz.html">Intel® Core™ i7-7820HQ Processor</a> and 32GB DDR4 RAM. Even with the same setup, your mileage may vary under different hardware configurations.</p>
<p>As it is quite a nuisance to write an article for every single purpose of the system, I will briefly cover all the roles of this machine, but not going into details. As a result, this article is <strong>not</strong> meant to be treated as a <em>system configuration guide</em>.</p>
<h2>The host operating system</h2>
<p>After quite long struggling with Arch Linux and Gentoo, especially <a href="https://jsteward.moe/install-gentoo-in-hyper-v.html">with the desktop experience</a>, I turned to the well-known Microsoft Windows operating system in 2018. After several iterations of reinstallation, the current installation has been serving since July 2019 (confirmed via battery report of OS). The specific system variant as of writing is shown as follows.</p>
<p><img alt="winver screenshot" src="/images/winver-20200225.png"></p>
<p>Note that the system is in Insiders Slow Ring to get a usable 20H1 build. This will be explained in the following section.</p>
<p>The host system functions as the primary interface (i.e. captures all the keyboard and mouse input from <em>me</em>, outputs to internal and external monitors, drives all connected peripherals, provides connectivity). Besides, it hosts almost all the complex development tools that require a GUI (e.g. VSCode, Visual Studio, IntelliJ IDEA, Vivado, IDA Pro, Intel Parallel Studio, etc.), utilities, the Chrome browser, Internet chat programs, Office and Adobe suites, as well as a little bit of leisure stuff (connects to the organization SMB share for some classical music). <a href="https://www.msys2.org/">MSYS2</a> is also installed for tasks that are largely platform agnostic but require a Unix utility to run (<a href="https://github.com/chipsalliance/rocket-chip/blob/8cec10850a217d49a34d24fc3ae799daed6bcf26/src/main/scala/diplomacy/DeviceTree.scala#L135">example</a>).</p>
<h2>Virtualization platforms</h2>
<p>You've seen it correct: the system has multiple virtualization platforms installed and running, namely Hyper-V and VMware Workstation. The two platforms would not work together if you're on current versions of Windows and VMware. According to <a href="https://t.me/mm256_cmpeq_epi8">imbushuo</a>, Microsoft has provided VMware with some internal APIs for VBS support. As a result, with <a href="https://blogs.vmware.com/workstation/2020/01/vmware-workstation-tech-preview-20h1.html">VMware Workstation Tech Preview 20H1</a> and <a href="https://blogs.windows.com/windowsexperience/2019/12/10/announcing-windows-10-insider-preview-build-19041/">Windows 10 Insider Preview Build 19041</a>, one can simultaneously enable the two platforms and boot virtual machines from both of them. However, due to the nature of nested virtualization (VMware workstation runs in the host domain of Hyper-V), a <a href="https://en.wikipedia.org/wiki/Second_Level_Address_Translation">performance impact</a> is anticipated, and VMware will hint you about this:</p>
<p><img alt="vmware hint" src="/images/vmware-nested-hint-20200225.png"></p>
<p>The two VM platforms host two different virtual machines, each serves different purposes. Both of the virtual machines are accessed over SSH via <a href="https://www.putty.org/">PuTTY</a>. Furthermore, home directories in the two VMs are mapped back into Windows via <a href="https://github.com/libfuse/sshfs">SSHFS</a> for convenient file exchanging.</p>
<ul>
<li>Hyper-V hosts an Arch Linux OS. This is currently the main development platform for Unix stuff like OCaml, Linux kernel, cross-compiling, $$\LaTeX$$, etc. The VM is <a href="https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/hyper-v-server/terminology">enlightened</a> for best I/O performance and memory balloon support.</li>
<li>VMware Workstation hosts an Arch Linux OS. This is for USB passthrough of the JTAG table to a Linux environment. Hyper-V does not support USB passthrough, yet <a href="http://openocd.org/">OpenOCD</a>, the embedded debug translator, on Windows require <a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/usbcon/winusb">WinUSB</a> drivers, which is incompatible with Vivado Hardware Server. The problem does not exist on Linux with <code>libusb-1.x</code>.</li>
</ul>
<p>With this setup, I can run OpenOCD whenever a gdb session is needed, and switch to <a href="https://aur.archlinux.org/packages/xilinx-hw-server/">hwserver</a> whenever I need to use Xilinx debug cores or program the FPGA.</p>Series on RISC-V on the Baidu Edgeboard ZU3EG2020-02-25T11:00:00+08:002021-02-20T19:29:00+08:00Pengcheng Xutag:jsteward.moe,2020-02-25:/edgeboard-series.html<p><img alt="RISC-V logo" src="/images/RISC-V-logo.svg"></p>
<h2>Intro</h2>
<p>I haven't been writing blog articles <a href="https://jsteward.moe/gsoc-2018-final-report.html">ever since August, 2018</a>. Under the strong request from <a href="https://t.me/jiegec">@jiegec</a>, I decided to record various related aspects of my recent works on <a href="https://github.com/KireinaHoro/rocket-zynqmp">booting untethered, standard Linux/RISC-V on a ZynqMP board</a>, forming a series of blog articles of which you can find links …</p><p><img alt="RISC-V logo" src="/images/RISC-V-logo.svg"></p>
<h2>Intro</h2>
<p>I haven't been writing blog articles <a href="https://jsteward.moe/gsoc-2018-final-report.html">ever since August, 2018</a>. Under the strong request from <a href="https://t.me/jiegec">@jiegec</a>, I decided to record various related aspects of my recent works on <a href="https://github.com/KireinaHoro/rocket-zynqmp">booting untethered, standard Linux/RISC-V on a ZynqMP board</a>, forming a series of blog articles of which you can find links below. The list may be updated if new, related articles come in.</p>
<h2>The articles</h2>
<p>This article serves mostly as a catalog for the coming up articles. If you're interested in <em>why</em> the whole idea has come up, read the section that follows.</p>
<ul>
<li><a href="https://jsteward.moe/dev-system-setup-edgeboard-series.html">Development System Setup - Edgeboard RISC-V Series</a></li>
<li><a href="https://jsteward.moe/zu3eg-purchase-edgeboard-series.html">Meet the Baidu Edgeboard ZU3EG - Edgeboard RISC-V Series</a></li>
<li><a href="https://jsteward.moe/risc-v-hardware-design-part-a-edgeboard-series.html">RISC-V Hardware Design: System & Block Design - Edgeboard RISC-V Series</a></li>
<li><a href="https://jsteward.moe/risc-v-hardware-design-part-b-edgeboard-series.html">RISC-V Hardware Design: Debug via BSCAN Chain - Edgeboard RISC-V Series</a></li>
<li><a href="https://jsteward.moe/risc-v-software-design-part-a-edgeboard-series.html">RISC-V Software Design: Bootloader - Edgeboard RISC-V Series</a></li>
<li><a href="https://jsteward.moe/risc-v-software-design-part-b-edgeboard-series.html">RISC-V Software Design: PS System - Edgeboard RISC-V Series</a></li>
</ul>
<h2>The idea</h2>
<p><em>Note: this section has not undergone thorough fact checks and may suffer from fallacies in factual accuracy or lack references. It is laid out here only for informative purposes.</em></p>
<p>The idea of running RISC-V on a ZynqMP dates back to 2019, when a project from <a href="https://ceca.pku.edu.cn/">the lab I'm currently in</a> required an experimental platform about accelerators coupling with RISC-V. The board we were holding at the time was a <a href="https://www.xilinx.com/products/boards-and-kits/ek-u1-zcu102-g.html">Xilinx ZCU102</a>. There had been <a href="https://github.com/li3tuo4/rc-fpga-zcu">a flow</a> adapted from the original work of <a href="https://github.com/ucb-bar/fpga-zynq">RocketChip on a Zynq</a> from UCB. Built around <a href="https://github.com/riscv/riscv-fesvr"><code>fesvr</code>, the FrontEnd SerVeR</a>, that project implemented a <strong>tethered</strong> system of RISC-V tied to the ARM core in a Zynq system via AXI. We will not cover too much the details of a <code>fesvr</code> system; briefly speaking, the RISC-V core uses DRAM from Zynq PS via AXI master port, and accepts external control via the <a href="https://github.com/ucb-bar/riscv-sodor/issues/13">UCB-specific, undocumented HTIF interface</a> over an AXI slave. This enabled various applications from ISA simulation to Linux boot.</p>
<p>While the idea of <code>fesvr</code> may be appealing to the academic society due to its simplicity and various features, the HTIF interface was never documented thoroughly, making it difficult to analyze and extend, let alone standardize. The <a href="https://riscv.org/wp-content/uploads/2016/01/Wed1115-untether_wsong83.pdf">Untethered lowRISC project</a> has enabled the Rocket cores to boot untethered-ly via the use of a <a href="https://github.com/riscv/riscv-pk">Berkeley Bootloader</a>. However, with the standardization of Machine-mode behavior by the <strong>RISC-V SBI specification</strong> and the release of the <a href="https://github.com/riscv/opensbi">reference OpenSBI implementation</a>, as well as the upstreaming of the GNU toolchain, U-Boot/RISC-V, and Linux/RISC-V, it's now time to follow the trend and build a new flow that adopts the standard paradigms of development on RISC-V.</p>Portage-powered Android: Gentoo GSoC 2018 Final Report2018-08-02T08:00:00+08:002018-08-02T08:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-08-02:/gsoc-2018-final-report.html<h2>Overview</h2>
<p>The project is still under active development. Due to the over-optimistic estimation of the time needed to work on the Android build system, not all the objectives originally planned in the proposal have been finished. The project itself, however, is going on smoothly and the work is expected to …</p><h2>Overview</h2>
<p>The project is still under active development. Due to the over-optimistic estimation of the time needed to work on the Android build system, not all the objectives originally planned in the proposal have been finished. The project itself, however, is going on smoothly and the work is expected to continue despite the finish of the GSoC program.</p>
<h2>Boot Gentoo system on Android hardware -- Preinit</h2>
<p>This part is based on the work of <a href="https://wiki.gentoo.org/wiki/Project:ARM64">the Gentoo ARM64 Project</a> with modifications to the boot sequence to properly boot on Android hardware. Due to the nature of a fixed bootloader for Android, the project <a href="https://github.com/KireinaHoro/preinit">Preinit</a> is created to handle early mount operations and launch Gentoo init with the embedded initramfs inside Android's in-house <code>boot.img</code> format. Designed to adapt the difference between various Android hardware, the project is extensible and easy to add support for new devices. An eselect module is offered for users to switch between device profiles or use their custom initramfs.</p>
<h2>Manage device kernel sources and headers with Portage</h2>
<p>Android devices use kernel sources that are heavily patched to drive the vendor-specific hardware, and the patches may never be submitted to the upstream Linux kernel source tree. Consequently, it is crucial to use the kernel source tree dedicated to the device. Ebuilds have been created in category <code>sys-kernel</code> in the <a href="https://gitweb.gentoo.org/proj/android.git/">proj/android.git</a> overlay.</p>
<p>Due to differences between the Android bootloader and regular bootloaders for GNU/Linux systems, a plugin to the kernel build system, <a href="https://github.com/KireinaHoro/installkernel">installkernel</a>, which serves to correctly install the kernel automatically, has been created. Installkernel packs the kernel and preinit files into a <code>boot.img</code> and flashes it to the correct partition.</p>
<p>Using the Android kernel headers matters when building Android components natively on the device. Ebuilds for kernel headers with the version corresponding to the kernel sources have been created in the <code>sys-kernel</code> category as well, and users are recommended to use the correct version of kernel headers.</p>
<h2>Start Android in a container for phone functionality</h2>
<p>The Android system is started as a guest in LXC in Gentoo. As the Android filesystem structure differs from GNU/Linux greatly and some paths overlap, keeping the Android filesystem tree in a separate place is more suitable rather than mixing them together. The project <a href="https://github.com/KireinaHoro/installkernel">SharkBait-setup</a> has been created to automatically set up a vanilla Android device with Gentoo chroot available as a Portage-powered Android system.</p>
<h2>Toolchain for Android system target</h2>
<p>Toolchains that run on different host architectures is important for building the Android system natively on the device. <a href="/toolchain-for-aarch64-linux-android.html">GCC</a> and <a href="/toolchain-clang-llvm-with-sanitiazers-for-android.html">LLVM</a> cross-compile toolchains for Android <code>aarch64-linux-android</code> target have been produced without Google's obscured scripts, which greatly helps in building Android on an AArch64 host--the Android device itself.</p>
<h2>Documentation Work</h2>
<p>Development details on specific topics can be found in my blog articles. These articles form the weekly reports of progress. The <a href="https://wiki.gentoo.org/wiki/User:Jsteward/SharkBait_User_Guide">user guide</a> and <a href="https://wiki.gentoo.org/wiki/User:Jsteward/SharkBait_User_Guide">porter's guide</a> have been created on Gentoo Wiki to serve as official documentation for the project.</p>
<h2>Community Building</h2>
<p>Regular discussion about the project takes place every Saturday at 2:00 AM UTC time in #shark-bait at Freenode IRC. Tyson Tan, the designer of the KDE mascot, have created a mascot and a logo for the project. A website <a href="https://www.shark-bait.org/">shark-bait.org</a> that serves as a portal and development blog has been created by fellow developers that helped me work on the project.</p>
<h2>Work left to be done</h2>
<p>The GSoC period is somewhat short to complete such a massive project to restructure Android, and many of the things that were originally planned are not finished yet. Hopefully they will eventually get implemented as the project evolves.</p>
<h3>Binary artifacts in the toolchain</h3>
<p>As a sensible way to build Bionic separately has not been worked out yet: the <code>libc</code> components in the toolchains created are copied from a live Android system. Proper build method of Bionic is required to avoid copying object files when creating the toolchain.</p>
<h3>Android Build system</h3>
<p>The manually-created toolchains have to be integrated into the Android build system to enable native building on AArch64. Many parts of the Android build system currently do not distinguish host OS and architecture, assuming that the host to be an x86 system. Work is needed to refactor these parts and enable the use of system toolchains instead of prebuilt ones.</p>
<h3>Dependency relationship between AOSP modules</h3>
<p>The AOSP sources are modularized, and incremental, reproducible building of the modules is supported. However, the dependency relationship between modules are implicit and dynamically resolved by the build system during the build. To build the modules separately, the dependency relationship needs to be explicitly figured out so that the dependencies are not built multiple times.</p>
<h3>Eclass for Android build system functions</h3>
<p>The Android build systems expose functions to build modules. The functions should be wrapped in an eclass to provide a consistent interface for ebuilds for Android components.</p>
<h3>Support for more devices</h3>
<p>Currently, the Portage-powered Android project only supports two devices: Huawei Nexus 6P and ASUS ZenFone 3 Ultra. Support for more and newer devices is expected to come in the future.</p>
<h2>Projects created during GSoC 2018</h2>
<ul>
<li><a href="https://github.com/KireinaHoro/preinit">Preinit</a></li>
<li><a href="https://github.com/KireinaHoro/installkernel">Installkernel</a></li>
<li><a href="https://github.com/KireinaHoro/sharkbait-setup">SharkBait-setup</a></li>
</ul>
<h2>Code and documentation merged upstream</h2>
<ul>
<li><a href="https://gitweb.gentoo.org/proj/android.git/log/">Commits in proj/gentoo.git</a></li>
<li>Pages on Gentoo Wiki:<ul>
<li><a href="https://wiki.gentoo.org/wiki/User:Jsteward/Starting_Android_in_LXC">Starting Android in LXC</a></li>
<li><a href="https://wiki.gentoo.org/wiki/User:Jsteward/Building_a_toolchain_for_aarch64-linux-android">Building a toolchain for aarch64-linux-android</a></li>
<li><a href="https://wiki.gentoo.org/wiki/User:Jsteward/Clang_toolchain_with_sanitizers_support_for_Android">Clang/LLVM toolchain with sanitizers support for Android</a></li>
<li><a href="https://wiki.gentoo.org/wiki/User:Jsteward/SharkBait_User_Guide">SharkBait User Guide</a></li>
<li><a href="https://wiki.gentoo.org/wiki/User:Jsteward/SharkBait_Porter%27s_Guide">SharkBait Porter's Guide</a></li>
</ul>
</li>
</ul>
<h2>GSoC reports archive</h2>
<p><a href="/category/gsoc-2018.html">Category GSoC 2018 on jsteward.moe</a></p>
<h2>Acknowledgment</h2>
<p>I appreciate the guidance from my mentor Benda Xu very much. He shows great passion and enthusiasm in my project while still gives me great freedom to determine how the project would go when issues occur. His extreme patience and timely help ensured the success of my GSoC 2018 project.</p>
<p>Stephen Christie and Lucas Ramage actively participated in the weekly discussion that takes place in the #shark-bait channel. They maintain the project website and provide crucial insights when I run into tough issues. Stephen also contacted the artist, Tyson Tan, who created the KDE mascot, to create a logo and a mascot for the project. Thanks for their devotion to the project. Also, I would like to show my gratefulness for Tyson for creating the cute, lovely mascot Mako. She exactly represents what we imagined about her.</p>
<p>I would like to express my gratitude for Bingxing Wang and Icenowy for helping me with the Linux kernel. Bingxing Wang helped me understand how the hardware and low-level software work on embedded ARM devices, while Icenowy provided important insights to handle problems I had encountered with the Android <code>boot.img</code> as well as knowledge with the GNU tools and ARM architecture.</p>
<p>I would also like to thank the Tsinghua University TUNA Association and its energetic members for providing me with a high-quality environment for technical discussions. Many times was I stuck an issue we heatedly discuss, and it gets resolved easily.</p>
<p>Finally, I would like to thank all the people who have encouraged, commented on, or criticized my project and everyone that worked on the project with me. Hope that our effort can make Android and Gentoo better platforms, and even better, the world a better place.</p>Gentoo GSoC Weekly Report 07/302018-07-29T21:00:00+08:002018-07-29T21:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-29:/weekly-report-0730.html<h2>Summary</h2>
<p>The last week was about writing guides and automatic setup scripts for users. The following package has been added to the overlay:</p>
<ul>
<li><code>sys-apps/sharkbait-setup</code>: setup for Android container on a SharkBait system</li>
</ul>
<p>The following articles have been added to the blog:</p>
<ul>
<li><a href="https://jsteward.moe/toolchain-clang-llvm-with-sanitiazers-for-android.html">Clang/LLVM toolchain with sanitizers support for Android …</a></li></ul><h2>Summary</h2>
<p>The last week was about writing guides and automatic setup scripts for users. The following package has been added to the overlay:</p>
<ul>
<li><code>sys-apps/sharkbait-setup</code>: setup for Android container on a SharkBait system</li>
</ul>
<p>The following articles have been added to the blog:</p>
<ul>
<li><a href="https://jsteward.moe/toolchain-clang-llvm-with-sanitiazers-for-android.html">Clang/LLVM toolchain with sanitizers support for Android</a></li>
<li><a href="https://jsteward.moe/sharkbait-user-guide.html">SharkBait User Guide</a></li>
<li><a href="https://jsteward.moe/sharkbait-porters-guide.html">SharkBait Porter's Guide</a></li>
</ul>
<p>All the repositories I've created for the SharkBait project now have a concise <code>README.md</code> now.</p>SharkBait Porter's Guide2018-07-27T23:00:00+08:002018-07-27T23:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-27:/sharkbait-porters-guide.html<h2>Welcome, porters!</h2>
<p>This article is intended for porters who want to add SharkBait support for a device that's not currently supported. In this article, you will be given the general idea on how to adapt parts that are crucial for SharkBait system to work on a device. The following components …</p><h2>Welcome, porters!</h2>
<p>This article is intended for porters who want to add SharkBait support for a device that's not currently supported. In this article, you will be given the general idea on how to adapt parts that are crucial for SharkBait system to work on a device. The following components will be covered:</p>
<ul>
<li>Preinit. <a href="https://github.com/KireinaHoro/preinit">repository home</a></li>
<li>SharkBait-setup. <a href="https://github.com/KireinaHoro/sharkbait-setup">repository home</a></li>
<li>Kernel sources. (<code>sys-kernel/${BOARD_NAME}-sources</code>) <a href="https://github.com/KireinaHoro/android/tree/master/sys-kernel/angler-sources">example</a></li>
<li>Kernel headers. (<code>sys-kernel/linux-headers-${BSP_VERSION}</code>) <a href="https://github.com/KireinaHoro/android/tree/master/sys-kernel/liunx-headers">example</a></li>
</ul>
<h2>Preinit</h2>
<p>Preinit performs early initialization of the device to load the Gentoo init (OpenRC as of current). The following directory structure is required for a supported device (<code>angler</code> as an example here) is as follows:</p>
<div class="highlight"><pre><span></span><code>angler
├── bootimg.cfg
├── initramfs
│ ├── init
│ └── (other contents in initramfs)
└── Makefile
</code></pre></div>
<ul>
<li><code>bootimg.cfg</code> defines offsets for kernel and initramfs, as well as kernel commandline options. Porters should adapt what they get when dissecting <code>boot.img</code> for their device and make the following modifications:<ul>
<li>remove <code>bootsize</code> option so that abootimg does not complain about a boot.img bigger than the original one (though this is unlikely to be the case).</li>
<li>add <code>androidboot.selinux=permissive</code> to the boot commandline to set SELinux to permissive.</li>
</ul>
</li>
<li><code>initramfs</code> holds the minimal initramfs that correctly mounts filesystems and <code>switch_root</code> to load Gentoo init.<ul>
<li><code>initramfs/init</code> is required. The <code>angler</code> init is a shell script that does the necessary jobs, but any executable file should work.</li>
<li>Mind the permissions: kernel won't be able to execute an <code>init</code> that's not executable, which would result in a boot failure.</li>
</ul>
</li>
<li><code>Makefile</code> enables processing the <code>initramfs</code> programmatically before <code>installkernel</code> packs it up.<ul>
<li>This will be useful for things like <code>busybox</code> installation, runtime-specific things, etc.</li>
<li><code>installkernel</code> will call <code>make</code> in the device directory (<code>angler</code> in the directory hierarchy). Read the example <code>Makefile</code> so that you handle the paths correctly.</li>
</ul>
</li>
</ul>
<h2>SharkBait-setup</h2>
<p>SharkBait-setup handles the setup of the Android container. The following directory structure is required for a supported device (<code>angler</code> as an example here) is as follows:</p>
<div class="highlight"><pre><span></span><code>angler
├── disable_encryption.sh
├── fstab.android
├── patches
│ ├── fstab.angler.patch
│ └── (other patches to apply to Android rootfs)
└── serial-consoles (if any)
</code></pre></div>
<ul>
<li><code>disable_encryption.sh</code> is ran on the helper workstation and disables encryption for the partition where the Gentoo root will reside in.<ul>
<li>For devices that does not support encryption or have the Gentoo root in a partition that's not encrypted, this script should just print a notice and return 0.</li>
<li><strong>Warn users about data wipe and wait for 10 seconds for a Ctrl-C!</strong></li>
</ul>
</li>
<li><code>fstab.android</code> is appended to the Gentoo fstab and contain mountpoints that are necessary for Android.<ul>
<li>Android's <code>vold</code> should not handle any internal mounts any more (external mounts such as OTG or SD card may still be managed by <code>vold</code>); make sure all required mounts are present.</li>
<li>It is strongly recommended to mount to <code>/var/lib/android</code> and then bind into the LXC rootfs for ease when accessing the Android partitions and extra security. Refer to the example <code>fstab.android</code> file for <code>angler</code> to get a better understanding of this.</li>
</ul>
</li>
<li><code>patches</code> holds patches that will be applied to the real Android rootfs via <code>patch -p0</code>, extracted from the current boot.img present on the device.<ul>
<li>Disable all partition mounts in <code>fstab.$DEVICE.patch</code> or equivalent file.</li>
<li>Put more patches that are required here, such as patches on <code>init.rc</code> to properly handle cgroups issues introduced by containerization. Refer to <code>init.rc.patch</code> for <code>angler</code> for more information on this topic.</li>
</ul>
</li>
<li><code>serial-consoles</code> defines serial consoles that are available on the device, if there is any. This file will be appended to <code>/etc/inittab</code>.</li>
</ul>
<h2>Kernel sources</h2>
<p>The kernel source package of a device packages the source tree from the vendor. A ebuild for <code>sys-kernel/${BOARD_NAME}-sources</code> is required; read the <a href="https://github.com/KireinaHoro/android/blob/master/sys-kernel/angler-sources/angler-sources-3.10.73.ebuild">ebuild for <code>angler-sources</code></a> for reference. Only the package name, version, and the git repo URL would require adapting; the other parts of the ebuild do not need modification.</p>
<p>The following modifications on the source tree are required for a working kernel:</p>
<ul>
<li>Apply <a href="https://github.com/KireinaHoro/android_kernel_huawei_angler/commit/be819350157b2aadcbc8db7001119130f0e51bad.patch">this patch</a> on the tree to enable <code>installkernel</code> function.</li>
<li>Supply a valid <code>defconfig</code> with LXC features enabled. See <a href="https://github.com/KireinaHoro/android_kernel_huawei_angler/blob/sharkbait/arch/arm64/configs/sharkbait_angler_defconfig">the example for angler</a> and <a href="https://wiki.gentoo.org/wiki/LXC#Kernel_options_required">LXC on Gentoo Wiki</a> for reference.</li>
<li>Make sure that the kernel <strong>compiles</strong> and <strong>boots correctly</strong> with (relatively) new compilers from Gentoo.</li>
<li>Regularly merge upstream changes.</li>
</ul>
<h2>Kernel headers</h2>
<p>Kernel headers that match the device kernel source ease the process of compiling cross-compile toolchains for Android targets. The <a href="https://github.com/KireinaHoro/android/blob/master/sys-kernel/linux-headers/linux-headers-3.10.73.ebuild">ebuild for <code>angler</code></a> should be a clear example, with only the need to modify the version number and the git repo URL.</p>
<h2>Test the port</h2>
<p>Make sure the port boots correctly. You may need a serial console to debug boot failures. Also, check that all the hardware functions work properly (camera, bluetooth, etc.); if that's not the case point it out when submitting a merge request so that we can look into the issue. Happy hacking!</p>SharkBait User Guide2018-07-25T19:00:00+08:002018-07-25T19:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-25:/sharkbait-user-guide.html<h2>Welcome to SharkBait!</h2>
<p>This article is intended for devices that are supported by SharkBait. The current list of supported devices is as follows:</p>
<ul>
<li>Huawei Nexus 6P (<code>angler</code>)</li>
</ul>
<p>If your device is not in the list, consult the <a href="https://jsteward.moe/sharkbait-porters-guide.html">Porter's Guide</a> for instructions on how to add support for a device. The …</p><h2>Welcome to SharkBait!</h2>
<p>This article is intended for devices that are supported by SharkBait. The current list of supported devices is as follows:</p>
<ul>
<li>Huawei Nexus 6P (<code>angler</code>)</li>
</ul>
<p>If your device is not in the list, consult the <a href="https://jsteward.moe/sharkbait-porters-guide.html">Porter's Guide</a> for instructions on how to add support for a device. The rest of this article assumes that you have a device that's supported. Let's get started!</p>
<h2>Install a normal Android system</h2>
<p>The first step is to install a normal version of Android on your device. Note that the system should have ADB root access (i.e. ability to <code>adb root</code> and get <code>adbd</code> running as root). <a href="https://lineageos.org/">Lineage OS</a> should be a good choice in many cases.</p>
<h2>Flash boot.img to disable full-disk encryption</h2>
<p>The supported devices have corresponding pages that contain links to scripts that modifies the <code>boot.img</code> extracted from the device to disable forced encryption. Clone <a href="https://github.com/KireinaHoro/sharkbait-setup">the SharkBait Setup repository</a> to your work computer and do the following.</p>
<p><strong>Warning:</strong> The following steps wipe your device's <code>userdata</code> and <code>cache</code> partition (a.k.a. "factory reset"). Make sure that important data is backed up before continuing.</p>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>sharkbait-setup
<span class="nv">DEVICE</span><span class="o">=</span>angler<span class="w"> </span><span class="c1"># substitute with your device name</span>
sudo<span class="w"> </span>./decrypt.sh<span class="w"> </span><span class="nv">$DEVICE</span>
</code></pre></div>
<p>After the device has boot up, verify that the following command doesn't output anything, which indicates that forced encryption has been disabled:</p>
<div class="highlight"><pre><span></span><code>adb<span class="w"> </span>shell<span class="w"> </span>mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"/dev/block/dm-"</span>
</code></pre></div>
<h2>Setup Gentoo chroot in /data/gnu</h2>
<p>Gain root access via adb. Make path <code>/data/gnu</code>; this will be the home of Gentoo root, which will become the system root soon. Follow <a href="https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage">Guide on Gentoo Wiki</a> to set up the Gentoo root. Note the following points during the setup:</p>
<ul>
<li>Pick <code>arm64</code> stage3 tarballs that can be found <a href="http://distfiles.gentoo.org/experimental/arm64/">here</a> instead of AMD64 stage3 tarballs;</li>
<li>Add user <code>portage</code> to gid 3003 (<code>inet</code>):<ul>
<li><code>groupadd -g 3003 inet && gpasswd -a portage inet</code></li>
</ul>
</li>
<li>Install <a href="https://gitweb.gentoo.org/proj/android.git/">the proj/android.git overlay</a> that contains Android-specific kernel sources and utilities. Refer to <a href="https://wiki.gentoo.org/wiki/Layman">Layman page on Gentoo Wiki</a> for how to add an overlay to the system.</li>
<li>Choose <code>sys-kernel/${BOARD_NAME}-sources</code> instead of the general <code>sys-kernel/gentoo-sources</code> when picking the kernel source. Install <code>virtual/android-sources</code> for packages that want the <code>virtual/linux-sources</code> virtual. Also, install <code>sys-kernel/linux-headers</code> with exactly the same version as the sources from the overlay.</li>
<li>Install <code>sys-kernel/installkernel</code> for automatic deploying boot.img on Android.<ul>
<li>Read post-installation message carefully.</li>
</ul>
</li>
<li>Configure the kernel as the following:<ul>
<li>Some kernels require a specific version of compiler. Try using an older version of compiler if problems occur (e.g. compile errors, unable to boot, etc.)</li>
<li><code>make sharkbait_${BOARD_NAME}_defconfig</code></li>
<li>change the kernel config according to your needs</li>
<li>try your best to config as builtin instead of modules due to poor module dependency system in Android kernel source</li>
<li><code>unset TMPDIR</code> before <code>make</code>: adb has strange environment variables that breaks build</li>
<li><strong>skip the installation of kernel</strong> (<strong>do not</strong> <code>make install</code> yet -- <code>sharkbait-deploy</code> needs the current boot.img to deploy Android container)</li>
</ul>
</li>
<li>Skip configuring the bootloader</li>
</ul>
<p>We need LXC to run Android in a container. The kernel sources of supported devices have necessary LXC options enabled by default. Install <code>sys-apps/sharkbait-setup</code> and do the following:</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEVICE</span><span class="o">=</span>angler<span class="w"> </span><span class="c1"># substitute with your device name</span>
sudo<span class="w"> </span>sharkbait-deploy<span class="w"> </span><span class="nv">$DEVICE</span>
</code></pre></div>
<h2>Install the kernel</h2>
<p>Now go to <code>/usr/src/linux</code> and install the kernel:</p>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>/usr/src/linux
make<span class="w"> </span>modules_install<span class="w"> </span><span class="o">&&</span><span class="w"> </span>make<span class="w"> </span>install
</code></pre></div>
<h2>Reboot and test!</h2>
<p>The genuine test for installing an operating system is rebooting and see if things work out as expected. Reboot the phone the Android way: select "Reboot" in the power menu. The phone should boot up normally. To go back into Gentoo, "dial home" via the shell script installed at <code>/data/ssh/dialhome</code>:</p>
<div class="highlight"><pre><span></span><code>adb<span class="w"> </span>shell
/data/ssh/dialhome
</code></pre></div>
<p>Join #shark-bait @ Freenode for discussion or for help in case anything goes wrong. Also, if you have other devices that are not yet supported by SharkBait, you can help with porting them, following the <a href="https://jsteward.moe/sharkbait-porters-guide.html">Porter's Guide</a>.</p>Clang/LLVM toolchain with sanitizers support for Android2018-07-25T16:00:00+08:002018-07-25T16:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-25:/toolchain-clang-llvm-with-sanitiazers-for-android.html<h2>Preface</h2>
<p>Toolchain work for Android has been progressing in the last few weeks, finally reaching a stage where a completely working Clang/LLVM toolchain is available. The toolchain can build test programs that run on vanilla AOSP as well as Lineage OS, with proper sanitizer support. Though actually plugging this …</p><h2>Preface</h2>
<p>Toolchain work for Android has been progressing in the last few weeks, finally reaching a stage where a completely working Clang/LLVM toolchain is available. The toolchain can build test programs that run on vanilla AOSP as well as Lineage OS, with proper sanitizer support. Though actually plugging this toolchain into Android build process is not tested yet, the sanitizers for the toolchain is built with the toolchain itself, and various problems in the headers has been fixed. This article explains complex parts in <a href="https://gist.github.com/KireinaHoro/282f6c1fef8b155126aaeb0acccf4280">the bootstrap script</a> as well as places where manual intervention is needed.</p>
<p>We're going to build pieces of the toolchain in this order to satisfy dependency:</p>
<ul>
<li>GCC toolchain (binutils, libc objects & headers, compiler)</li>
<li>LLVM + Clang</li>
<li>Compiler-rt builtins</li>
<li>Libunwind</li>
<li>Libc++abi</li>
<li>Libc++</li>
<li>Compiler-rt non-builtins (sanitizers, profilers, etc.)</li>
</ul>
<h2>Set up environment and repositories</h2>
<p>This part defaults to using <code>aarch64-linux-android</code> as the target and using the HEAD version of LLVM tools. Edit accordingly if this is not desired. CMake and Ninja is required to follow this guide; install them if they're not present on your system.</p>
<h2>Set up GCC for target</h2>
<p>GCC is needed for cross compiling with Clang/LLVM. Follow <a href="https://jsteward.moe/toolchain-for-aarch64-linux-android.html">this article on GCC cross-compiling for Android</a> to get a working copy of cross GCC. This should get you through the comment block that reads:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># install GNU binutils</span>
<span class="c1"># copy headers into $PREFIX/$TARGET/sys-include</span>
<span class="c1"># copy crt*.o lib{c,m,dl}.so to $PREFIX/lib64 and $PREFIX/$TARGET/lib</span>
<span class="c1"># install GCC</span>
</code></pre></div>
<p>Note that later Clang/LLVM build requires prebuilt libraries in two different locations. Copy the object files accordingly.</p>
<h2>LLVM + Clang</h2>
<p>Configure options to note:</p>
<ul>
<li><code>LLVM_TARGETS_TO_BUILD=AArch64</code>: Enable only AArch64 target. Remember to substitute this if the target is not <code>aarch64</code> (e.g. <code>ARM</code> for 32 bit)</li>
</ul>
<h2>Compiler-rt builtins</h2>
<p>Configure options to note:</p>
<ul>
<li><code>CMAKE_INSTALL_PREFIX=$PREFIX/lib/clang/7.0.0</code>: Install to Clang "resource path" so that libraries can be automatically found by Clang while compiling.</li>
<li><code>COMPILER_RT_BUILD_BUILTINS=ON</code> and <code>COMPILER_RT_BUILD_*=OFF</code>: Build builtins only as other components require <code>libc++</code> to build.</li>
</ul>
<h2>Libunwind</h2>
<p>Nothing special here.</p>
<h2>Libcxxabi</h2>
<p>Configure options to note:</p>
<ul>
<li><code>LIBCXXABI_LIBCXX_INCLUDES="../../libcxx/include"</code>: Specify libc++ header path for reference by libc++abi during build.</li>
<li><code>LIBCXXABI_USE_COMPILER_RT=ON</code>: This option name speaks for itself.</li>
<li><code>LIBCXXABI_USE_LLVM_UNWINDER=ON</code>: This option name speaks for itself.</li>
</ul>
<p>Remember to install libc++abi headers into the prefix as CMake doesn't generate libc++abi header install rules:</p>
<div class="highlight"><pre><span></span><code>sudo<span class="w"> </span>cp<span class="w"> </span>-R<span class="w"> </span>../include<span class="w"> </span><span class="nv">$PREFIX</span>/include/libcxxabi
</code></pre></div>
<h2>Libcxx</h2>
<p>As discussed in <a href="https://reviews.llvm.org/D46558">this thread</a>, LLVM HEAD at the time of writing uses NDK headers that are newer than the prebuilt NDK version (r16 vs r14). As a result, we have to rebase to remove commit <code>85a7702b4cc5d69402791fe685f151cf3076be71</code> from Libcxx repository:</p>
<div class="highlight"><pre><span></span><code><span class="nb">pushd</span><span class="w"> </span>..<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>git<span class="w"> </span>fetch<span class="w"> </span>--unshallow<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>git<span class="w"> </span>rebase<span class="w"> </span>-i<span class="w"> </span>85a7702b4cc5d69402791fe685f151cf3076be71^<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nb">popd</span>
</code></pre></div>
<p>Configure options to note:</p>
<ul>
<li><code>LIBCXX_CXX_ABI="libcxxabi"</code>: Use <code>libcxxabi</code> as the C++ ABI library (instead of <code>libcxxrt</code> or <code>libsupc++</code>).</li>
<li><code>LIBCXX_CXX_ABI_INCLUDE_PATHS="$PREFIX/include/libcxxabi"</code>: Reference to <code>libcxxabi</code> headers (installed in the previous step).</li>
<li><code>LIBCXX_USE_COMPILER_RT=ON</code>: This option name speaks for itself.</li>
</ul>
<p>Note that Android by default combines libc++, libc++abi, and libunwind into a single <code>libc++.so</code>. We'll do this as well so that the executable does not contain stray dynamic link references.</p>
<div class="highlight"><pre><span></span><code><span class="nb">pushd</span><span class="w"> </span><span class="nv">$TARGET</span>/lib
sudo<span class="w"> </span>mv<span class="w"> </span>libc++.so<span class="w"> </span>libc++.so.old
sudo<span class="w"> </span>/usr/local/aarch64-linux-android/bin/clang<span class="w"> </span>-shared<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-o<span class="w"> </span>libc++.so<span class="w"> </span>-Wl,--whole-archive<span class="w"> </span>libc++.a<span class="w"> </span>libc++abi.a<span class="w"> </span>libunwind.a<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-Wl,--no-whole-archive
<span class="nb">popd</span>
</code></pre></div>
<h2>Compiler-rt non-builtin (sanitizers, etc.)</h2>
<p>Bionic headers as of commit <code>a9713035baecf21f607ef81c8652eb344086966c</code> misses definition for <code>in_addr_t</code> in its headers. It is possible that the Linux headers are expected to define this, but <code>android_kernel_huawei_angler</code> did not define this. Apply <a href="https://gist.github.com/KireinaHoro/141d27321b2aab27fa8292b1bd0f7105">this patch</a> on Bionic headers to continue.</p>
<p>Configure options to note:</p>
<ul>
<li><code>CMAKE_INSTALL_PREFIX=$PREFIX/lib/clang/7.0.0</code>: Install to Clang "resource path" so that libraries can be automatically found by Clang while compiling.</li>
<li><code>COMPILER_RT_BUILD_BUILTINS=OFF</code> and <code>COMPILER_RT_BUILD_*=ON</code>: Build components other than builtins as we have <code>libc++</code> now.</li>
</ul>
<h2>Testing</h2>
<div class="highlight"><pre><span></span><code><span class="c1"># C:</span>
/usr/local/aarch64-linux-android/bin/clang<span class="w"> </span>hello.c<span class="w"> </span>-o<span class="w"> </span>hello
<span class="c1"># C++:</span>
/usr/local/aarch64-linux-android/bin/clang++<span class="w"> </span>hello.cc<span class="w"> </span>-o<span class="w"> </span>hello<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--stdlib<span class="o">=</span>libc++<span class="w"> </span>--rtlib<span class="o">=</span>compiler-rt
<span class="c1"># C++ with sanitizers (ubsan as an example):</span>
/usr/local/aarch64-linux-android/bin/clang++<span class="w"> </span>hello.cc<span class="w"> </span>-o<span class="w"> </span>hello<span class="w"> </span>--stdlib<span class="o">=</span>libc++<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-fsanitize<span class="o">=</span>undefined<span class="w"> </span>-static-libsan<span class="w"> </span>--rtlib<span class="o">=</span>compiler-rt
</code></pre></div>Gentoo GSoC Weekly Report 07/232018-07-22T00:00:00+08:002018-07-22T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-22:/weekly-report-0723.html<h2>Summary</h2>
<p>I continued to work on toolchains for <code>-android</code> and <code>-androideabi</code> targets in the past week. The following articles summarize the work done:</p>
<ul>
<li><a href="https://jsteward.moe/toolchain-for-aarch64-linux-android.html">Building a toolchain for aarch64-linux-android</a> describes how to build a toolchain for target <code>aarch64-linux-android</code> and compile <strong>dynamically linked</strong> executables that run on <strong>unmodified</strong> Android platform.</li>
<li><a href="https://gist.github.com/KireinaHoro/282f6c1fef8b155126aaeb0acccf4280">Bootstrap aarch64-linux-android …</a></li></ul><h2>Summary</h2>
<p>I continued to work on toolchains for <code>-android</code> and <code>-androideabi</code> targets in the past week. The following articles summarize the work done:</p>
<ul>
<li><a href="https://jsteward.moe/toolchain-for-aarch64-linux-android.html">Building a toolchain for aarch64-linux-android</a> describes how to build a toolchain for target <code>aarch64-linux-android</code> and compile <strong>dynamically linked</strong> executables that run on <strong>unmodified</strong> Android platform.</li>
<li><a href="https://gist.github.com/KireinaHoro/282f6c1fef8b155126aaeb0acccf4280">Bootstrap aarch64-linux-android Clang/LLVM toolchain</a> is a shell script that creates a working Clang/LLVM toolchain that cross-compiles for Android target.</li>
</ul>
<p>The final evaluation of GSoC is nearing, and I will begin the work on the final report for evaluation, merging my work in <code>android.git</code>, and tidying up blog posts to post on the Gentoo Wiki. Things planned that are not finished yet:</p>
<ul>
<li>Bionic build (and ebuilds) without AOSP tree</li>
<li>Ebuilds for GCC toolchain for Android (or integrate into crossdev; requires Bionic to be properly handled)</li>
<li>AOSP build on AArch64 (with toolchain drop-in or directly use system toolchain)</li>
<li>Per-component build (implemented by Android build system already; requires native build on AArch64 done to start)</li>
</ul>
<p>This checklist may go beyond GSoC 2018 time span, but I hope that the end of GSoC doesn't mark the end of the Portage-powered Android project. Also, please don't hesitate to contact me if you want to help (and are not sure where to start)! You can reach me by the means in my <a href="https://jsteward.moe/pages/about-and-contact.html">Contact Page</a>.</p>
<p>Besides the ongoing development of building Android on AArch64 host and dissecting AOSP, I am starting to work on wrap-up articles that provide steps that users (well, not <em>end users</em>) as well as porters can follow to reproduce existing work easier. The following articles are planned:</p>
<ul>
<li>Set up Portage-powered Android -- User Guide</li>
<li>Add a new device -- Porter's Guide</li>
</ul>Building a toolchain for aarch64-linux-android2018-07-21T01:00:00+08:002018-07-21T01:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-21:/toolchain-for-aarch64-linux-android.html<h2>Preface</h2>
<p>To build the Android Platform, we need a toolchain that can produce executables that link correctly against Android's <code>libc</code>, Bionic. To achieve this, a toolchain targeting <code>aarch64-linux-android</code> (and <code>arm-linux-androideabi</code>) is needed. This article aims to provide a step-by-step guide for reproducing the desired toolchain on all architecture that GCC …</p><h2>Preface</h2>
<p>To build the Android Platform, we need a toolchain that can produce executables that link correctly against Android's <code>libc</code>, Bionic. To achieve this, a toolchain targeting <code>aarch64-linux-android</code> (and <code>arm-linux-androideabi</code>) is needed. This article aims to provide a step-by-step guide for reproducing the desired toolchain on all architecture that GCC supports running on, including AArch64, which my GSoC project needs. Unfinished work on this topic is documented at the end of this article.</p>
<h2>Step 0 -- Clone source and set up environment variables</h2>
<p>Clone the following repositories which hold Google's modifications to the GCC toolchain for Android target:</p>
<ul>
<li><a href="https://android.googlesource.com/toolchain/binutils">Binutils</a></li>
<li><a href="https://android.googlesource.com/platform/bionic">Bionic</a></li>
<li><a href="https://android.googlesource.com/toolchain/binutils">GCC</a></li>
</ul>
<p>The rest of this article assumes that <code>$PWD</code> is in the corresponding project for each step.</p>
<p>Export the environment variables to get the paths right:</p>
<div class="highlight"><pre><span></span><code><span class="nb">export</span><span class="w"> </span><span class="nv">TARGET</span><span class="o">=</span>aarch64-linux-android
<span class="nb">export</span><span class="w"> </span><span class="nv">PREFIX</span><span class="o">=</span>/usr/local/<span class="nv">$TARGET</span>
</code></pre></div>
<h2>Step 1 -- Build and install binutils</h2>
<p>Binutils that can process binaries for the target is needed. Create a separate build directory, configure, compile, and then install for target:</p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>build<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>build
../binutils-2.27/configure<span class="w"> </span>--target<span class="o">=</span><span class="nv">$TARGET</span><span class="w"> </span>--prefix<span class="o">=</span><span class="nv">$PREFIX</span><span class="w"> </span>--enable-werror<span class="o">=</span>no
make<span class="w"> </span>-j8<span class="w"> </span><span class="o">&&</span><span class="w"> </span>sudo<span class="w"> </span>make<span class="w"> </span>install
</code></pre></div>
<p>The <code>--enable-werror=no</code> option is used to work around failed compiles due to newer versions of GCC generating new warnings. Examine <code>$PREFIX</code> and see if the binutils for the target has been properly installed.</p>
<h2>Step 2 -- Install prebuilt libc & headers</h2>
<p>The next step, as most cross-compiler creation guides instructs, is to install libc. Unfortunately, we have not figured out a proper way to compile Bionic without Android's (gigantic) build system, so we're just doing a voodoo copy-and-paste. The libc part in this section is expected to get better when we develop a mature solution for building Bionic.</p>
<p>Install <code>sys-kernel/linux-headers-3.10.73</code> from <a href="https://github.com/KireinaHoro/android/tree/master/sys-kernel/linux-headers">here</a>. Set up libc and kernel headers.</p>
<div class="highlight"><pre><span></span><code>sudo<span class="w"> </span>emerge<span class="w"> </span>-av<span class="w"> </span><span class="o">=</span>sys-kernel/linux-headers-3.10.73
mkdir<span class="w"> </span>-p<span class="w"> </span><span class="nv">$PREFIX</span>/<span class="nv">$TARGET</span>/sys-include
cp<span class="w"> </span>-Rv<span class="w"> </span>bionic/libc/include/*<span class="w"> </span><span class="nv">$PREFIX</span>/<span class="nv">$TARGET</span>/sys-include/
<span class="k">for</span><span class="w"> </span>a<span class="w"> </span><span class="k">in</span><span class="w"> </span>linux<span class="w"> </span>asm<span class="w"> </span>asm-generic<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span>ln<span class="w"> </span>-s<span class="w"> </span>/usr/include/<span class="nv">$a</span><span class="w"> </span><span class="nv">$PREFIX</span>/<span class="nv">$TARGET</span>/sys-include/
<span class="k">done</span>
</code></pre></div>
<p>The following object files are needed for a successful generation of the toolchain:</p>
<ul>
<li><code>crtbegin_so.o</code>: from NDK</li>
<li><code>crtend_so.o</code>: from NDK</li>
<li><code>crtbegin_dynamic.o</code>: from NDK</li>
<li><code>crtend_android.o</code>: from NDK</li>
<li><code>libc.so</code>: from AOSP</li>
<li><code>libm.so</code>: from AOSP</li>
<li><code>libdl.so</code>: from AOSP</li>
<li><code>ld-android.so</code>: from AOSP</li>
</ul>
<p>Obtain the above files and place them under <code>$PREFIX/$TARGET/lib</code> for discovery by the linker.</p>
<h2>Step 3 -- Build and install GCC</h2>
<p>The final part is relatively simple. Just compile GCC and install it into the toolchain prefix:</p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>build<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">cd</span><span class="w"> </span>build
../gcc-4.9/configure<span class="w"> </span>--target<span class="o">=</span><span class="nv">$TARGET</span><span class="w"> </span>--prefix<span class="o">=</span><span class="nv">$PREFIX</span><span class="w"> </span>--without-headers<span class="w"> </span>--with-gnu-as<span class="w"> </span>--with-gnu-ld<span class="w"> </span>--enable-languages<span class="o">=</span>c,c++
make<span class="w"> </span>-j8<span class="w"> </span><span class="o">&&</span><span class="w"> </span>sudo<span class="w"> </span>make<span class="w"> </span>install
</code></pre></div>
<h2>Step 4 -- Build and verify "Hello, world!"</h2>
<p>We need to verify that the toolchain is really working by creating executables for our target. Write a simple "Hello, world!" program and compile it with:</p>
<div class="highlight"><pre><span></span><code><span class="nv">aarch64</span><span class="o">-</span><span class="nv">linux</span><span class="o">-</span><span class="nv">android</span><span class="o">-</span><span class="nv">g</span><span class="o">++</span><span class="w"> </span><span class="nv">hello</span>.<span class="nv">cc</span><span class="w"> </span><span class="o">-</span><span class="nv">o</span><span class="w"> </span><span class="nv">hello</span><span class="w"> </span><span class="o">-</span><span class="nv">pie</span><span class="w"> </span><span class="o">-</span><span class="nv">static</span><span class="o">-</span><span class="nv">libgcc</span><span class="w"> </span><span class="o">-</span><span class="nv">nostdinc</span><span class="o">++</span><span class="w"> </span><span class="o">-</span><span class="nv">I</span><span class="o">/</span><span class="nv">usr</span><span class="o">/</span><span class="nv">local</span><span class="o">/</span><span class="nv">aarch64</span><span class="o">-</span><span class="nv">linux</span><span class="o">-</span><span class="nv">android</span><span class="o">/</span><span class="k">include</span><span class="o">/</span><span class="nv">c</span><span class="o">++/</span><span class="nv">v1</span><span class="w"> </span><span class="o">-</span><span class="nv">nodefaultlibs</span><span class="w"> </span><span class="o">-</span><span class="nv">lc</span><span class="w"> </span><span class="o">-</span><span class="nv">lm</span><span class="w"> </span><span class="o">-</span><span class="nv">lc</span><span class="o">++</span>
</code></pre></div>
<p>Explanation for the commandline options used:</p>
<ul>
<li><code>-pie</code>: Android requires <a href="https://en.wikipedia.org/wiki/Position-independent_code">Position Independent Executables</a> property for dynamically-linked executables.</li>
<li><code>-static-libgcc</code>: Android platform does not have <code>libgcc_s.so</code> available; we'll have to make it statically-linked.</li>
<li><code>-nostdinc++</code> and <code>-I...</code>: Android uses <code>libc++</code> as its default STL implementation, and we need this to get the right symbols used by including right C++ headers.<ul>
<li>This suggests that a correct copy of <code>libcxx</code> headers should be present at the path shown above.</li>
</ul>
</li>
<li><code>-nodefaultlibs</code> and <code>-l...</code>: by default GCC links to <code>libstdc++</code>, which is not desirable in this case; we manually specify what to consider during the linking process.<ul>
<li>This suggests that <code>libc++.so</code> should be present in the linker search path.</li>
</ul>
</li>
</ul>
<h2>What else?</h2>
<p>The work is not finished yet on this topic:</p>
<ul>
<li>We still copy-and-paste <code>libc</code> object files instead of properly building them separately. Proper packaging of <code>bionic</code> is needed.</li>
<li>The toolchain build process needs integrating with <code>crossdev</code>, Gentoo's flexible cross-compile toolchain generator.</li>
</ul>Using Yubikey as a login token2018-07-18T15:00:00+08:002018-07-18T15:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-18:/yubikey-as-login-token.html<h2>The story</h2>
<p>It has been a while since <a href="https://jsteward.moe/meet-yubikey.html">I got my Yubikey from Harry Chen</a>. Now that I'm back working on Gentoo, so it is time to explore functions of the Yubikey on Linux. Instead of just using the Yubikey as a OpenPGP card, its <a href="https://en.wikipedia.org/wiki/Universal_2nd_Factor">U2F</a> function can serve more …</p><h2>The story</h2>
<p>It has been a while since <a href="https://jsteward.moe/meet-yubikey.html">I got my Yubikey from Harry Chen</a>. Now that I'm back working on Gentoo, so it is time to explore functions of the Yubikey on Linux. Instead of just using the Yubikey as a OpenPGP card, its <a href="https://en.wikipedia.org/wiki/Universal_2nd_Factor">U2F</a> function can serve more than simply logging you into Google. In this article, we'll show how to make the Yubikey a convenient and secure login token for Linux systems, enabling you to log into the system with a plug and a touch, and lock the session when the token is removed.</p>
<h2>U2F capability for PAM</h2>
<p>U2F can be used locally on Linux systems thanks to the <a href="https://github.com/Yubico/pam-u2f"><code>pam_u2f</code> module by Yubico</a>. Emerge <code>sys-auth/pam_u2f</code> and add the active user (<code>jsteward</code> in my case) to the <code>usb</code> group for access to the device:</p>
<div class="highlight"><pre><span></span><code>emerge<span class="w"> </span>-av<span class="w"> </span>sys-auth/pam_u2f
gpasswd<span class="w"> </span>-a<span class="w"> </span>jsteward<span class="w"> </span>usb
</code></pre></div>
<p><strong>Note:</strong> remember to log out and then log back in to make the new group settings come into place.</p>
<h3>Register the device</h3>
<p>Make sure you've configured the appropriate kernel options (<code>CONFIG_HIDRAW</code> and <code>CONFIG_USB_HIDDEV</code>). Plug in the device and observe <code>dmesg</code> to see if the deivce is detected by the kernel correctly. We can then setup the authorization mappings, registers the device to the user that uses it.</p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>-p<span class="w"> </span>~/.config/Yubico
pamu2fcfg<span class="w"> </span>-ujsteward<span class="w"> </span>><span class="w"> </span>~/.config/Yubico/u2f_keys
</code></pre></div>
<p>Touch the device as the LED on it starts blinking. Verify that the device is properly registered by verifying the contents of <code>~/.config/Yubico/u2f_keys</code>. To use multiple keys, register the device with different authorization file locations (i.e. <code>u2f_keys</code>) and then merge them into a single <code>u2f_keys</code> in the following format:</p>
<div class="highlight"><pre><span></span><code>jsteward:<KeyHandle1>,<UserKey1>:<KeyHandle2>,<UserKey2>:...
</code></pre></div>
<h3>Configure PAM module</h3>
<p>We want to use our Yubikey for system authentications, which would include logins (DE or tty), session unlock (screensaver), <code>sudo</code>, and more. Place the following line in <code>/etc/pam.d/system-auth</code>:</p>
<div class="highlight"><pre><span></span><code>auth sufficient pam_u2f.so cue
</code></pre></div>
<p>Just above the following existing line in the file:</p>
<div class="highlight"><pre><span></span><code>auth required pam_unix.so try_first_pass likeauth nullok
</code></pre></div>
<p><strong>Note:</strong> the <code>cue</code> parameter makes the module display a prompt "Please touch the device." when it is waiting for a response so that the user does not mistake it as the system has hung.</p>
<p>This enables you to authenticate with a Yubikey without the need of user passwords. To use the Yubikey as a 2FA tool, change <code>sufficient</code> to <code>required</code>. However, this will result in failed authentication when the Yubikey is not present (and touched in time).</p>
<p><strong>Warning:</strong> <code>system-auth</code> affects all authentications, including remote ones. Think twice before using <code>required</code> for <code>pam_u2f</code> on a server.</p>
<p><strong>Warning:</strong> messing up PAM configuration may result in being locked out of the system, which is only possible to fix with a LiveCD or <code>init=/bin/bash</code>.</p>
<p>Test the configuration by trying to log in or sudo. SDDM logins will be a little bit tricky due to the way it is designed: press Login without entering a correct password and touch the device.</p>
<h2>Lock the session when Yubikey gets removed</h2>
<p>This works with a simple <code>udev</code> rule. For an easy way of locking sessions, use <code>elogind</code> instead of the default <code>consolekit</code> (on Gentoo) as described in the <a href="https://wiki.gentoo.org/wiki/Elogind">Gentoo Wiki page for <code>elogind</code></a>. Write the following rule in <code>/etc/udev/rules.d/20-yubikey.rules</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">ACTION</span><span class="o">==</span><span class="s">"remove"</span><span class="p">,</span> <span class="n">ENV</span><span class="p">{</span><span class="n">ID_BUS</span><span class="p">}</span><span class="o">==</span><span class="s">"usb"</span><span class="p">,</span> <span class="n">ENV</span><span class="p">{</span><span class="n">ID_MODEL_ID</span><span class="p">}</span><span class="o">==</span><span class="s">"0407"</span><span class="p">,</span> <span class="n">ENV</span><span class="p">{</span><span class="n">ID_VENDOR_ID</span><span class="p">}</span><span class="o">==</span><span class="s">"1050"</span><span class="p">,</span> <span class="n">RUN</span><span class="o">+=</span><span class="s">"/bin/loginctl lock-sessions"</span>
</code></pre></div>
<p>Check your Yubikey's VID:PID pair and substitute <code>1050:0407</code> accordingly. Reload the udev rules:</p>
<div class="highlight"><pre><span></span><code><span class="n">sudo</span><span class="w"> </span><span class="n">udevadm</span><span class="w"> </span><span class="n">control</span><span class="w"> </span><span class="o">--</span><span class="n">reload</span><span class="o">-</span><span class="n">rules</span>
</code></pre></div>
<p>Unplug your Yubikey to see the rule in action.</p>Gentoo GSoC Weekly Report 07/162018-07-16T05:00:00+08:002018-07-16T05:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-16:/weekly-report-0716.html<h2>Summary</h2>
<p>This week is mainly about decoupling toolchain from Android Build System so as to enable a local build of Android on the phone itself. Toolchains for the following programming languages are needed throughout the entire build:</p>
<ul>
<li>C/C++ (Android Platform)</li>
<li>Java (Android Platform, apps)</li>
<li>Go (build system, i.e …</li></ul><h2>Summary</h2>
<p>This week is mainly about decoupling toolchain from Android Build System so as to enable a local build of Android on the phone itself. Toolchains for the following programming languages are needed throughout the entire build:</p>
<ul>
<li>C/C++ (Android Platform)</li>
<li>Java (Android Platform, apps)</li>
<li>Go (build system, i.e. Soong & Blueprint)</li>
</ul>
<p>As of current, the entire source tree is mounted onto the phone via NFS. Tweak kernel config to enable NFS, emerge <code>net-fs/nfs-utils</code>, then mount the filesystem. Refer to <code>nfs(5)</code> for more information. The articles in this report assume that the AOSP tree is mounted at <code>/import/lineageos</code>.</p>
<p>The last week's work sums up into the following two articles:</p>
<ul>
<li><a href="https://jsteward.moe/aarch64-build-host.html">Enabling aarch64 as a host architecture for Android Build (WIP)</a>: logs modifications made to accomodate system toolchains for <code>aarch64</code> into the build system.<ul>
<li>This is necessary as Google doesn't provide prebuilt toolchains for <code>aarch64</code> hosts; the build system assumes <code>x86</code> here and there as well.</li>
</ul>
</li>
<li><a href="https://jsteward.moe/distcc-speed-up.html">Using distcc to speed up builds on phone</a>: speed up Portage builds on phone with <code>distccd</code> running on the laptop.</li>
<li><a href="https://jsteward.moe/toolchain-for-android.html">C/C++ Toolchain for Android Build on aarch64 (WIP)</a>: attempt to get the set of toolchains for building Android on aarch64.</li>
</ul>
<p>This week did not achieve any significant milestones; instead, several obstacles are discovered. They are discussed in the relative articles.</p>Using distcc to speed up builds on phone2018-07-16T02:00:00+08:002018-07-16T02:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-16:/distcc-speed-up.html<h2>Preface</h2>
<p>Compiling is the main source of time consumption on a Gentoo system, and Portage-powered Android is of no exception. Gentoo packages can leverage <a href="https://wiki.gentoo.org/wiki/Distcc">Distcc</a> to significantly speed up builds on slow machines with the aid of a powerful machine. For utilization in the Portage-powered Android project, we need to …</p><h2>Preface</h2>
<p>Compiling is the main source of time consumption on a Gentoo system, and Portage-powered Android is of no exception. Gentoo packages can leverage <a href="https://wiki.gentoo.org/wiki/Distcc">Distcc</a> to significantly speed up builds on slow machines with the aid of a powerful machine. For utilization in the Portage-powered Android project, we need to enable cross-compiling support for distcc. This article is a digest of the following two Gentoo Wiki articles, concisely logging the process to use distcc properly:</p>
<ul>
<li><a href="https://wiki.gentoo.org/wiki/Distcc">Distcc</a></li>
<li><a href="https://wiki.gentoo.org/wiki/Distcc/Cross-Compiling">Distcc/Cross-Compiling</a></li>
</ul>
<h2>Setup on the machine running <code>emerge</code></h2>
<p><code>Distcc</code> functions as a client handing out jobs to servers on the machine running <code>emerge</code>. Emerge distcc and Portage's distcc support, where <code>$REMOTE_CORES</code> is the number of cores available on the remote helper machine:</p>
<div class="highlight"><pre><span></span><code>emerge<span class="w"> </span>sys-devel/distcc
<span class="nv">N</span><span class="o">=</span><span class="k">$(($(</span>nproc<span class="k">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nv">$REMOTE_CORES</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m">1</span><span class="k">))</span>
<span class="nv">M</span><span class="o">=</span><span class="k">$(($(</span>nproc<span class="k">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m">2</span><span class="k">))</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"MAKEOPTS=\"-j</span><span class="nv">$N</span><span class="s2"> -l</span><span class="nv">$M</span><span class="s2">\"</span>
<span class="s2">FEATURES=\"distcc\""</span><span class="w"> </span>>><span class="w"> </span>/etc/portage/make.conf
</code></pre></div>
<p>Then configure distcc itself. As distcc does not use authentication (which would significantly slow down builds), it is recommended to have the build machine and the emerge machine in the same LAN and deploy proper firewall rules or ACLs. Run the following and substitute <code>192.168.1.1</code> with the IP address of your helper machine:</p>
<div class="highlight"><pre><span></span><code>distcc-config --set-hosts "192.168.1.1"
</code></pre></div>
<p>A special mode called "pump mode" for distcc which offloads the preprocessing work to the helper box may also be used, which may even speed up the build speed. In some cases (e.g. <code>sys-devel/llvm</code>) the pump server may fail to determine which headers to send to the helper box, so it may need to be disabled in some occasions. To enable pump mode:</p>
<div class="highlight"><pre><span></span><code>distcc-config --set-hosts "192.168.1.1,cpp,lzo"
sed -i -e 's/^FEATURES="distcc"$/FEATURES="distcc distcc-pump"/' /etc/portage/make.conf
</code></pre></div>
<p>As what distcc does is proxying compiler calls to the helper server, and we're doing cross-compile here, we need to correctly set up a wrapper script for compiler symlinks that calls the correct <code>CHOST</code>. Put the following script into <code>/usr/local/sbin/distcc-fix</code> and give it executable permission:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1"># Clang aware, now your >chromium-65 ebuilds will use distcc just like before ;)</span>
<span class="c1"># We extract $TUPLE from make.conf to avoid editing the script for each architecture.</span>
<span class="nv">TUPLE</span><span class="o">=</span><span class="k">$(</span>portageq<span class="w"> </span>envvar<span class="w"> </span>CHOST<span class="k">)</span>
<span class="nv">GCC_VER</span><span class="o">=</span><span class="k">$(</span>gcc-config<span class="w"> </span>-c<span class="p">|</span>cut<span class="w"> </span>-d<span class="w"> </span><span class="s2">"-"</span><span class="w"> </span>-f5<span class="k">)</span>
<span class="nv">CLANG_VER</span><span class="o">=</span><span class="k">$(</span>clang<span class="w"> </span>--version<span class="p">|</span>grep<span class="w"> </span>version<span class="p">|</span>cut<span class="w"> </span>-d<span class="w"> </span><span class="s2">" "</span><span class="w"> </span>-f3<span class="p">|</span>cut<span class="w"> </span>-d<span class="s1">'.'</span><span class="w"> </span>-f1,2<span class="k">)</span>
<span class="nb">cd</span><span class="w"> </span>/usr/lib/distcc/bin
rm<span class="w"> </span>cc<span class="w"> </span>c++<span class="w"> </span>gcc<span class="w"> </span>g++<span class="w"> </span>gcc-<span class="si">${</span><span class="nv">GCC_VER</span><span class="si">}</span><span class="w"> </span>g++-<span class="si">${</span><span class="nv">GCC_VER</span><span class="si">}</span><span class="w"> </span>clang<span class="w"> </span>clang++<span class="w"> </span>clang-<span class="si">${</span><span class="nv">CLANG_VER</span><span class="si">}</span><span class="w"> </span>clang++-<span class="si">${</span><span class="nv">CLANG_VER</span><span class="si">}</span><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper
<span class="nb">echo</span><span class="w"> </span><span class="s1">'#!/bin/bash'</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper
<span class="nb">echo</span><span class="w"> </span><span class="s2">"exec </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span><span class="s2">-g\${0:\$[-2]}"</span><span class="w"> </span><span class="s2">"\"\$@\""</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper
<span class="nb">echo</span><span class="w"> </span><span class="s1">'#!/bin/bash'</span><span class="w"> </span>><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper
<span class="nb">echo</span><span class="w"> </span><span class="s2">"exec </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span><span class="s2">-\$(basename \${0}) \"\$@\""</span><span class="w"> </span>>><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper
chmod<span class="w"> </span><span class="m">755</span><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper
chmod<span class="w"> </span><span class="m">755</span><span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span>cc
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span>c++
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span>gcc
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span>g++
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span>gcc-<span class="si">${</span><span class="nv">GCC_VER</span><span class="si">}</span>
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-wrapper<span class="w"> </span>g++-<span class="si">${</span><span class="nv">GCC_VER</span><span class="si">}</span>
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper<span class="w"> </span>clang
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper<span class="w"> </span>clang++
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper<span class="w"> </span>clang-<span class="si">${</span><span class="nv">CLANG_VER</span><span class="si">}</span>
ln<span class="w"> </span>-s<span class="w"> </span><span class="si">${</span><span class="nv">TUPLE</span><span class="si">}</span>-clang-wrapper<span class="w"> </span>clang++-<span class="si">${</span><span class="nv">CLANG_VER</span><span class="si">}</span>
</code></pre></div>
<p>Do the following to update the symlinks for distcc and enable automatic symlink update in case of distcc/gcc/clang upgrade:</p>
<div class="highlight"><pre><span></span><code>/usr/local/sbin/distcc-fix
cat<span class="w"> </span>>><span class="w"> </span>/etc/portage/bashrc<span class="w"> </span><span class="s"><< EOF</span>
<span class="s">case ${CATEGORY}/${PN} in</span>
<span class="s"> sys-devel/distcc | sys-devel/gcc | sys-devel/clang)</span>
<span class="s"> if [ "${EBUILD_PHASE}" == "postinst" ]; then</span>
<span class="s"> /usr/local/sbin/distcc-fix &</span>
<span class="s"> fi</span>
<span class="s"> ;;</span>
<span class="s">esac</span>
<span class="s">EOF</span>
</code></pre></div>
<h2>Setup on the helper machine</h2>
<p>Distcc runs in server mode here. Emerge <code>sys-devel/distcc</code>, and setup the cross-compile toolchain as described in <a href="https://jsteward.moe/building-lxc-ready-kernel.html">this article</a> if you haven't done it yet.</p>
<p>Enable <code>distccd</code> service on the helper box. Edit <code>/etc/conf.d/distccd</code> and add <code>--allow 192.168.1.0/24</code> to <code>DISTCCD_OPTS</code> to allow your device to send jobs. You may also want to configure detailed logging to debug, and separate the log file so that it does not pollute the syslog. The configuration file comes with descriptive comments for reference.</p>
<h2>Verify</h2>
<p>Install something via <code>emerge</code>. Observe if there is any improvement in build time. Observe <code>distccd</code> logs on the helper machine to see if any error happens. More usages can be found in <a href="https://wiki.gentoo.org/wiki/Distcc#Usage">the wiki page for distcc</a>.</p>C/C++ Toolchain for Android Build on aarch64 (WIP)2018-07-16T00:00:00+08:002018-07-16T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-16:/toolchain-for-android.html<h2>Disclaimer</h2>
<p>This article is a <em>stub</em>. This means that it may only serve status report purposes; it may contain severe scientific fallacies and/or inappropriate language, or have missing sections of information. In case of disagreement on previous point, <strong>cease consumption/utilization of content in this webpage immediately</strong>.</p>
<h2>Preface</h2>
<p>Many …</p><h2>Disclaimer</h2>
<p>This article is a <em>stub</em>. This means that it may only serve status report purposes; it may contain severe scientific fallacies and/or inappropriate language, or have missing sections of information. In case of disagreement on previous point, <strong>cease consumption/utilization of content in this webpage immediately</strong>.</p>
<h2>Preface</h2>
<p>Many essential components in the Android platform are written in C/C++ and, apparently, toolchainss are required to build them correctly. Google designed Android's build system to only run on <code>linux-x86</code> or <code>darwin-x86</code>, and they shipped prebuilt toolchains for only x86, <code>x86_64-unknown-linux-gnu</code> to be specific. To build for an <code>arm64</code> device on a <code>x86_64</code> host, toolchains targeting the following <a href="https://wiki.osdev.org/Target_Triplet">target triplets</a> are needed:</p>
<ul>
<li><code>x86_64-linux-glibc</code>: for compiling build tools that run on the host</li>
<li><code>aarch64-linux-android</code>: for compiling <code>arm64</code> objects that run on the device <em>in 64bit</em></li>
<li><code>arm-linux-androideabi</code>: for compiling <code>arm</code> objects that run on the device <em>in 32bit</em></li>
</ul>
<p>All the toolchains above run on the build host (<code>x86_64</code> in a standard build). What we need to do is to make these toolchains available on <code>aarch64</code>, while not changing the target. Note that <code>x86_64-linux-glibc</code> produces objects for <em>the build host</em>, so it should actually be the host's native compiler: if we're building on <code>aarch64</code>, it should be something like <code>aarch64-unknown-linux-gnu</code>.</p>
<p>Most parts of Android uses Clang/LLVM to build as well. We'll need the respective targets for Clang/LLVM on the <code>aarch64</code> build host as well. We'll look into this later in this article.</p>
<h2>GCC part</h2>
<h3>Host toolchain: <code>aarch64-unknown-linux-gnu</code></h3>
<p>This is the simplest part: Gentoo's system compiler should <em>just work</em>. What we need to do here is to find out the "root" of the host toolchain, just like the path for the prebuilt toolchains as recorded in <a href="https://github.com/LineageOS/android_build_soong"><code>build/soong</code></a>. We can then mimic how the prebuilt x86 toolchains get registered in the build system to inject our native toolchain into the system.</p>
<h3>Device toolchain: <code>aarch64-linux-android</code> & <code>arm-linux-androideabi</code></h3>
<p>These toolchains are cross-compile toolchains that produce code for the Android platform. The difference between these toolchains and a normal GNU/Linux toolchain for <code>aarch64</code> is that a normal toolchain uses Glibc, and the Android toolchain uses <a href="https://en.wikipedia.org/wiki/Bionic_(software)">Bionic</a> for libc. Experiments on building a Android GCC toolchain that runs on <code>aarch64</code> are being conducted at present and is still in its beginning.</p>
<h2>Clang/LLVM part</h2>
<p>Clang is used to compile most of the C/C++ code in the Android platform; Google is even compiling the Linux kernel with Clang (<a href="https://github.com/nathanchance/android-kernel-clang">guide on GitHub</a>), so apparently a Clang/LLVM toolchain for the Android target is necessary. A brief description of building the clang toolchain in the AOSP source tree can be found in <a href="https://hardenedlinux.github.io/toolchains/2016/04/01/How_to_build_Clang_toolchains_for_Android.html">this article (external)</a>, but obviously more work is needed to build for <code>aarch64</code> hosts, and integrating the results into the Android build system for the new build host.</p>
<h2>Integrating the toolchains into the build system</h2>
<p>Unfortunately, the Android system is strongly coupled with its prebuilt toolchains, and the paths for toolchains and architecture detection is littered around the repo (mainly in the build/ repositories). Known locations of toolchain path references: (paths below are clickable)</p>
<ul>
<li><a href="https://android.googlesource.com/platform/build/+/master/envsetup.sh">build/make/envsetup.sh</a></li>
<li><a href="https://github.com/LineageOS/android_build/blob/lineage-15.1/core/envsetup.mk">build/make/core/envsetup.mk</a></li>
<li><a href="https://github.com/LineageOS/android_build_soong/blob/lineage-15.1/Android.bp">build/soong/Android.bp</a></li>
<li><a href="https://github.com/LineageOS/android_build_soong/blob/lineage-15.1/cc/config/x86_linux_host.go">build/soong/cc/config/x86_linux_host.go</a></li>
</ul>
<p>The process of integrating the toolchain configured in the sections above into the build system is a painstaking one, as Google hardly documents the intrinsic mechanisms of the gigantic, complex build system. A series of trial-and-error attempts is expected.</p>Enabling aarch64 as a host architecture for Android Build (WIP)2018-07-15T12:00:00+08:002018-07-15T12:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-15:/aarch64-build-host.html<h2>Disclaimer</h2>
<p>This article is a <em>stub</em>. This means that it may only serve status report purposes; it may contain severe scientific fallacies and/or inappropriate language, or have missing sections of information. In case of disagreement on previous point, <strong>cease consumption/utilization of content in this webpage immediately</strong>.</p>
<h2>Preface</h2>
<p>As …</p><h2>Disclaimer</h2>
<p>This article is a <em>stub</em>. This means that it may only serve status report purposes; it may contain severe scientific fallacies and/or inappropriate language, or have missing sections of information. In case of disagreement on previous point, <strong>cease consumption/utilization of content in this webpage immediately</strong>.</p>
<h2>Preface</h2>
<p>As a key step in the Portage-powered Android project, we need the ability to build Android on an <code>aarch64</code> host (not necessarily Android, as Gentoo which is beneath Android is essentially a normal aarch64 host now) so that we can build the system on the phone itself. However, the current Android build system assumes one of the following types of build hosts:</p>
<ul>
<li>Linux on AMD64</li>
<li>Darwin on AMD64</li>
</ul>
<p>We need to add <code>aarch64</code> and <code>arm</code> (32bit variant) toolchain support into the build system. Google originally provieded toolchain support by shipping prebuilt toolchains with the AOSP project; apparently we have to use local toolchains or build cross-compile toolchains on our own. This article logs modifications to the AOSP build system (<code>${T}/build</code>) to accomodate system toolchain into the build system. All source modifications can be found in my <a href="https://github.com/KireinaHoro?tab=repositories">forked repositories</a>, which all came from Lineage OS project originally.</p>
<h1>Get the build system to recognize aarch64 as a valid HOST arch</h1>
<p>The repository <a href="https://github.com/KireinaHoro/android_build"><code>build/make</code></a> holds core makefiles that describe how the build system works. The following code in <code>build/make/core/envsetup.mk</code> defines the <code>HOST_ARCH</code> section for the build host:</p>
<div class="highlight"><pre><span></span><code><span class="c"># HOST_ARCH</span>
<span class="cp">ifneq (,$(findstring x86_64,$(UNAME)))</span>
<span class="w"> </span>HOST_ARCH<span class="w"> </span>:<span class="o">=</span><span class="w"> </span>x86_64
<span class="w"> </span>HOST_2ND_ARCH<span class="w"> </span>:<span class="o">=</span><span class="w"> </span>x86
<span class="w"> </span>HOST_IS_64_BIT<span class="w"> </span>:<span class="o">=</span><span class="w"> </span><span class="nb">true</span>
<span class="cp">else</span>
<span class="cp">ifneq (,$(findstring i686,$(UNAME))$(findstring x86,$(UNAME)))</span>
<span class="nf">$(error Building on a 32-bit x86 host is not supported</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">UNAME</span><span class="k">)</span>!)
<span class="cp">endif</span>
<span class="cp">endif</span>
</code></pre></div>
<p>Add <code>aarch64</code> as a host architecture and <code>arm</code> as its secondary architecture to make the build system aware that <code>aarch64</code> is a valid <code>HOST</code> for building.</p>
<h2>Use host Go toolchain</h2>
<p>Google has shipped prebuilt Go toolchains for <code>linux-x86</code> and <code>darwin-x86</code>, just like they did with C/C++. We'll use the system toolchain from Portage instead. Emerge <code>dev-lang/go</code>, then edit the toolchain paths in <a href="https://github.com/KireinaHoro/android_build_soong"><code>build/soong</code></a>, specifically <code>GOROOT</code>, to utilize the system toolchain:</p>
<h3>TODO: source code quote here</h3>
<h2>Use system Ninja for building</h2>
<p>Google shipped Ninja as well. Emerge <code>dev-util/ninja</code> and then edit the following in <a href="#stub"><code>build/blueprint</code></a> to use system ninja:</p>
<h3>TODO: source code quote here</h3>
<h2>Install Java toolchain</h2>
<p>An Android build requires a Java toolchain to be installed. Google does not bundle it this time: we're required to install it on the system, and the build system will try to locate it automatically at runtime. Fortunately, Gentoo has provided a binary package (!!) for OpenJDK (<code>icedtea</code> in this case), named <code>dev-java/icedtea-bin</code>. We just have to install this package. To avoid unnecessary dependencies getting pulled in, configure the use flags for <code>dev-java/icedtea-bin</code> as follows:</p>
<div class="highlight"><pre><span></span><code>dev-java/icedtea-bin headless-awt -alsa -cups -gtk -webstart
</code></pre></div>
<p>Then emerge <code>virtual/jdk</code>. More information can be found on <a href="https://wiki.gentoo.org/wiki/Java">the Java page on Gentoo Wiki</a>.</p>
<h2>Configure C/C++ toolchain</h2>
<p>The hardest part is actually the C/C++ toolchain. The topic is discussed in a separate article: <a href="https://jsteward.moe/toolchain-for-android.html">C/C++ Toolchain for Android Build on aarch64 (WIP)</a>. The article briefly introduces toolchains involved in the building process. It then describes means to get them work locally on <code>aarch64</code>. Finally, the article describes how to accomodate the toolchains with <code>aarch64</code> as the host architecture into the Soong build system (path <code>build/soong/cc/config</code> to be specific).</p>Gentoo GSoC Weekly Report 07/092018-07-08T00:00:00+08:002018-07-08T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-08:/weekly-report-0709.html<h2>Summary</h2>
<p>This is my first week back on GSoC from the final exams, and, according to the proposal, the second part of the project should get started in this week. This week's work sums up into the following two articles:</p>
<ul>
<li><a href="https://jsteward.moe/android-kernel-source-portage.html">Integrating Android kernel source into Portage</a>
Introduces how the three …</li></ul><h2>Summary</h2>
<p>This is my first week back on GSoC from the final exams, and, according to the proposal, the second part of the project should get started in this week. This week's work sums up into the following two articles:</p>
<ul>
<li><a href="https://jsteward.moe/android-kernel-source-portage.html">Integrating Android kernel source into Portage</a>
Introduces how the three components, <code>preinit</code>, <code>installkernel</code>, and <code>${BOARD_NAME}-sources</code> works together to provide a kernel source package that enables users to build and install kernels just like normal Gentoo Linux systems on a Portage-powered Android system.</li>
<li><a href="https://jsteward.moe/android-build-system.html">Brief introduction to the (post-8.0) Android Build System</a>
Summarizes the information I collected via searching and reading sources in AOSP about the current Android build system that came into action at present (Android 8.0 and later). Attempts to dissect the build system also included.</li>
</ul>
<p>And, as Part I (as the fundamentals for Portage-powered Android) is now (almost) finished (what's left would be to package LXC-related and OpenRC-related things up as a package), I believe that it's time to announce the official short-and-sweet name for the Portage-powered Android project: <strong>SharkBait</strong>! Despite I'm the person that does most of the coding work (at least during the GSoC session), SharkBait actually has a team under the hood, and they're extremely helpful whenever I get stuck, and provide excellent ideas. A site is available <a href="https://www.shark-bait.org">here</a>, and despite not much is there at present, it'll definitely grow with time. We'll have a logo soon as well, so stay tuned!</p>Brief introduction to the (post-8.0) Android Build System2018-07-05T04:00:00+08:002018-07-05T04:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-05:/android-build-system.html<h2>Preface</h2>
<p>As the main target of the second period of my GSoC 2018 project, I have to take down Android build system and accomodating it into Portage, forming a modular building and updating mechanism. So, inevitably, I'll have to deal with the original AOSP build system (Lineage OS's is a …</p><h2>Preface</h2>
<p>As the main target of the second period of my GSoC 2018 project, I have to take down Android build system and accomodating it into Portage, forming a modular building and updating mechanism. So, inevitably, I'll have to deal with the original AOSP build system (Lineage OS's is a little bit different, but the core hasn't changed). In this article, I'll document the information I obtained from the Internet and (a tiny portion of) AOSP source about the build system. I'll then propose possible solutions to take AOSP apart.</p>
<h2>Present Android build system -- <code>Android.bp</code></h2>
<p>If one look at the repository structure of AOSP, he will find many components structured as separate repositories, which together form the AOSP tree. Inside each repository, code falls into smaller structures called "modules", with each module as a fundamental unit in dependency management and building the process (yeah, it's a complicated system).</p>
<p>Each module has its own build configuration file: <code>Android.mk</code> and/or <code>Android.bp</code>.
* <code>Android.mk</code> is the good, old format used everywhere by Android, from system components to application packages; it gets parsed into <a href="https://ninja-build.org/">Ninja</a> files by the <a href="https://github.com/google/kati">Kati</a> project.
* <code>Android.bp</code> is the new format that gets parsed by <a href="https://github.com/google/blueprint/">Blueprint</a>, and then converted to <a href="https://ninja-build.org/">Ninja</a> files though detailed build logics defined by <a href="https://android.googlesource.com/platform/build/soong/">Soong</a>. (Actually in AOSP, Blueprint is a part of Soong.)</p>
<p>Developers start building by sourcing the <code>build/envsetup.sh</code> script, which sets up necessary environemntal variables as well as some useful functions:</p>
<ul>
<li><code>breakfast</code>: specifies the product to build for.</li>
<li><code>lunch</code>: start a full build for the previously picked product.</li>
<li><code>brunch</code>: (as its name suggests) the combined effect of <code>breakfast</code> and <code>lunch</code>.</li>
<li><code>m</code>: makes from the top of the tree.</li>
<li><code>mm</code>: builds all of the modules in the current directory.</li>
<li><code>mmm</code>: builds all of the modules in the supplied directories.</li>
<li><code>croot</code>: go back to the root of source tree.</li>
</ul>
<p>The output will be stored at <code>out/target/product/${DEVICE}/</code>, while this path will be stored in the environmental variable <code>$OUT</code>.</p>
<h2>Experiments with the build system</h2>
<p>AOSP builds are <a href="https://reproducible-builds.org/">reproducible</a>: that is to say, as long as the source doesn't change, the produced binaries are identical. This was verified by comparing the hash of components already installed on my Nexus 6P with the ones I compile again from the source. With that, there turns out to be two approaches possible to achieve the goal of breaking Android apart:</p>
<ul>
<li>Parse the <code>Android.mk</code> and <code>Android.bp</code> files, resolving the relationship between components, and build the components with Portage<ul>
<li>current problem: despite that <code>Android.bp</code> files are simple, the build rules are unclear (Soong had very little documentation and is prone to change <em>at any time</em>); <code>Android.mk</code> files, on the other hand, can be very complicated: they're full-blown GNU Makefiles.</li>
</ul>
</li>
<li>Leave Android build system untouched (mostly) and produce executables by building modules <em>in</em> the build system. We then pack up each of the modules' outputs, removing dependencies from the output, and generating binary packages.<ul>
<li>current problem: the exact dependency relationship will be difficult, if possible, to find out.</li>
</ul>
</li>
</ul>
<p>It's difficult to say that which route will be picked, and I hope that this week's discussion should shed some light on this somehow tough problem.</p>Integrating Android kernel source into Portage2018-07-05T02:00:00+08:002018-07-05T03:30:00+08:00Pengcheng Xutag:jsteward.moe,2018-07-05:/android-kernel-source-portage.html<h2>Introduction</h2>
<p>The Linux kernel is the most important component on any systems that are based on it, be it Gentoo Linux or Android. The user may want to tweak the kernel to enable functions that userspace utilities need (e.g. LVM, FUSE, Netfilter, etc.), so being able to tweak the …</p><h2>Introduction</h2>
<p>The Linux kernel is the most important component on any systems that are based on it, be it Gentoo Linux or Android. The user may want to tweak the kernel to enable functions that userspace utilities need (e.g. LVM, FUSE, Netfilter, etc.), so being able to tweak the configurations and install a new version easily is important. Android's kernel, however, defaults to a pretty locked-down version with few standard Linux features besides the ones that Android needs on by default, and the kernel is tied with the initramfs into an Android-specific format called a <code>boot.img</code>. This article introduces how Linux kernel compiling and installation works on normal Gentoo Linux systems; then it documents how <code>preinit</code>, a custom <code>installkernel</code>, as well as the device-specific <code>${DEVICE}-sources</code> (<code>angler-sources</code> in case of Nexus 6P) works together; finally, it describes how to port other devices' kernel sources using the same model.</p>
<h2>On a normal system</h2>
<p>On a normal Gentoo Linux system, <code>sys-kernel/*-sources</code> holds the kernel sources, each variant with some differences (mostly in patchsets): <code>gentoo-sources</code> is officially supported by the Gentoo Kernel Team, while <code>ck-sources</code> includes Cons Koliva's kernel patchset. The source tree gets installed to <code>/usr/src/linux-*</code>, which then gets symlink'ed to <code>/usr/src/linux</code>. The user can install multiple kernel source trees on the system at the same time, while the active <code>/usr/src/linux</code> symlink is maintained by the <code>kernel</code> module of <code>eselect</code>.</p>
<p>To compile and install a kernel, the user first picks a suitable kernel source package, then goto <code>/usr/src/linux</code> and tweak options as he would with <code>make menuconfig</code> or other means. He then builds the kernel and installs it via <code>make install</code>, which calls <code>/sbin/installkernel</code> provided by <code>sys-apps/debianutils[kernel_linux]</code>. <code>installkernel</code> installs the produced kernel to <code>/boot</code> along with the kernel config and <code>System.map</code>, perserving the last kernel as a backup. Finally, it runs <code>run-parts</code>, which is also from <code>sys-apps/debianutils</code>, to run any post-install hooks (e.g. LILO entry update).</p>
<h2>On a Portage-powered Android system</h2>
<p>The Android kernel sources are no different in form factor from normal kernel sources: they have the same build system with normal Linux kernels; the only difference is that <code>make install</code> should work differently than regular Linux systems: Android phones expect <code>boot.img</code> in the boot partition instead of kernel images in <code>/boot</code>. In order to satisfy this requirement, the following plan that consists of three components is developed:</p>
<ul>
<li><a href="https://github.com/KireinaHoro/preinit"><code>sys-kernel/preinit</code></a>: offers device-specific initramfs files and <code>boot.img</code> parameters (kernel & initramfs offsets and boot commandline)</li>
<li><a href="https://github.com/KireinaHoro/installkernel"><code>sys-kernel/installkernel</code></a>: installs the correct kernel image (with dtb on ARM) to <code>/boot</code>, creates <code>boot.img</code>, and flashing it to the correct partition</li>
<li><a href="https://github.com/KireinaHoro/android_kernel_huawei_angler"><code>sys-kernel/${DEVICE}-sources</code></a>: kernel source with necessary patches and <code>defconfig</code> to work on a Portage-powered system</li>
</ul>
<p>Preinit offers an <code>eselect</code> module for choosing the device's model. The package's <code>pkg_postinst</code> will try to detect the current device via <code>androidboot.hardware</code> value in <code>/proc/cmdline</code>; if it failed to get a match, the user would have to manually select a device for the <code>preinit</code> files, or use the <code>custom</code> device (in which the user manually implement the preinit files).</p>
<p>The above three components have their ebuilds available <a href="https://github.com/KireinaHoro/android/tree/master/sys-kernel">in the <code>sys-kernel</code> category in this overlay</a>. Add the overlay and emerge <code>sys-kernel/${DEVICE}-sources</code>; Portage will automatically pull in the dependencies. After that, one can install kernel just like on a Linux system:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/usr/src/linux
$<span class="w"> </span>make<span class="w"> </span><span class="si">${</span><span class="nv">DEFCONFIG</span><span class="si">}</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span>make<span class="w"> </span>-j<span class="k">$(($(</span>nproc<span class="k">)</span><span class="o">+</span><span class="m">1</span><span class="k">))</span>
$<span class="w"> </span>sudo<span class="w"> </span>make<span class="w"> </span>modules_install
$<span class="w"> </span>sudo<span class="w"> </span>make<span class="w"> </span>install
</code></pre></div>
<p>The <code>boot.img</code> will be automatically created and flashed to the boot partition.</p>
<h2>Porting guide</h2>
<p>To add new device support to this framework, the following work needs to be done:</p>
<ul>
<li>Preinit:<ul>
<li>Implement a minimal initramfs, with a <code>/init</code> that mounts the necessary filesystems and launches OpenRC <code>init</code>. Implement the initramfs building logic (e.g. busybox symlinking) in a <code>Makefile</code>.</li>
<li>Dissect existing <code>boot.img</code> for the device with <code>abootimg</code>, producing a <code>bootimg.cfg</code>. Remove the <code>bootsize</code> parameter so that <code>abootimg</code> doesn't complain when the new image is bigger.</li>
<li><a href="https://github.com/KireinaHoro/preinit/tree/master/devices/angler">This example for angler</a> should guide you through the port.</li>
</ul>
</li>
<li><code>${DEVICE}-sources</code>:<ul>
<li>Create a <code>defconfig</code> that is LXC-capable. Refer to <a href="https://jsteward.moe/building-lxc-ready-kernel.html">the previous blog</a> for details.</li>
<li>Apply <a href="https://github.com/KireinaHoro/android_kernel_huawei_angler/commit/be819350157b2aadcbc8db7001119130f0e51bad?diff=unified">this patch</a> on the kernel sources. This is needed to enable our custom <code>installkernel</code>.</li>
<li>Create <code>sys-kernel/${DEVICE}-sources</code> ebuild. Follow <a href="https://github.com/KireinaHoro/android/blob/master/sys-kernel/angler-sources/angler-sources-3.10.73.ebuild">this</a> as an example.</li>
</ul>
</li>
</ul>Gentoo GSoC Weekly Report 05/282018-05-26T15:00:00+08:002018-05-26T15:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-26:/weekly-report-0528.html<h2>Summary</h2>
<p>This week was planned as the last week for the first (Preparation) and second (Gaining control of the system) parts in my <a href="https://docs.google.com/document/d/1v3yA4rkex5DGiPmdlXOSse5QZrZAL8QDJFNY2fFAMRY/edit">GSoC 2018 proposal</a>. I'm pleased to announce that I've successfully hit the target as planned: a video demonstration posted on Twitter can be watched <a href="https://twitter.com/KireinaHoro/status/1000328318497902593">here</a>. </p>
<p>Reading and …</p><h2>Summary</h2>
<p>This week was planned as the last week for the first (Preparation) and second (Gaining control of the system) parts in my <a href="https://docs.google.com/document/d/1v3yA4rkex5DGiPmdlXOSse5QZrZAL8QDJFNY2fFAMRY/edit">GSoC 2018 proposal</a>. I'm pleased to announce that I've successfully hit the target as planned: a video demonstration posted on Twitter can be watched <a href="https://twitter.com/KireinaHoro/status/1000328318497902593">here</a>. </p>
<p>Reading and following the instructions in the following articles in order would form a complete, detailed tutorial to reproduce the work up to this point:</p>
<ul>
<li>Article written in the week before this week:<ul>
<li><a href="https://jsteward.moe/building-gentoo-chroot-in-android.html">Building a Gentoo chroot in Android</a></li>
</ul>
</li>
<li>Articles I've written this week:<ul>
<li><a href="https://jsteward.moe/nexus-6p-uart.html">UART on Nexus 6P</a></li>
<li><a href="https://jsteward.moe/booting-gentoo-on-nexus-6p.html">Booting Gentoo on Nexus 6P</a></li>
<li><a href="https://jsteward.moe/building-lxc-ready-kernel.html">Buidling a LXC-ready Android kernel with Gentoo toolchain</a></li>
<li><a href="https://jsteward.moe/starting-android-in-lxc.html">Starting Android in LXC</a></li>
</ul>
</li>
</ul>
<p>Important repositories (excl. experimental ones that do not mean much) created up to this point (the repository names are clickable):</p>
<ul>
<li><a href="https://github.com/KireinaHoro/android_device_huawei_angler"><code>android_device_huawei_angler</code></a>: holds patch to disable the forced Full-Disk Encryption (as described in the "What's Next?" section in <a href="https://jsteward.moe/analysis-of-android-cryptfs.html">this article</a>);</li>
<li><a href="https://github.com/KireinaHoro/preinit_angler"><code>preinit_angler</code></a>: holds the <code>preinit</code>, initramfs structure, and Makefile rules to easily unpack and repack Android <code>boot.img</code>;</li>
<li><a href="https://github.com/KireinaHoro/android-lxc-files"><code>android-lxc-files</code></a>: holds patches, LXC config, startup and stop hooks, and helper scripts for the proper functioning of Android in LXC;</li>
<li><a href="https://github.com/KireinaHoro/android_kernel_huawei_angler"><code>android_kernel_huawei_angler</code></a>: holds the modified version of Android kernel source along with config tweaks to build a LXC-ready kernel with Gentoo cross-compile toolchain.</li>
</ul>
<p>The above repositories still lack proper <code>README</code>s, and I put that as the main task for part 4 (Cleaning up) in my GSoC project, whose main goal is to create proper documentation for future pickup.</p>
<p>We still lack some sort of distribution / release method, but I think that belongs to the goals in the third part of the project (Taking Android apart), as we'll have stage3 tarballs and automatic setup scripts available directly at the end of that part of work.</p>
<p>We need to modify Android framework to enable it to inform the host about its states. We also need some sort of interface (e.g. an app) in the Android world to perform actions outside the container (e.g. initialize system updates, switch LXC profiles, etc.) as we proceed further in part 3 of this project. Some other goals yet to achieve are recorded in the "Known Problems" section in <em><a href="https://jsteward.moe/starting-android-in-lxc.html">Starting Android in LXC</a></em>.</p>
<h2>Plans for the coming weeks</h2>
<p>June is drawing near, and it's time for my final-term examinations in the university. Just as planned in the time schedule, I'll pause all activities of GSoC during June, except filling the form for First Evaluation at around June 15, and restoring to full work capacity at the end of June. Looking forward to the third part (Taking Android apart) of the project. Thanks for everyone that devoted time to help me through the second part of the project--it was very fun and I learned a lot in the process.</p>
<p>Happy hacking!</p>Starting Android in LXC2018-05-26T13:00:00+08:002018-05-26T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-26:/starting-android-in-lxc.html<h2>Preface</h2>
<p>After successfully booting Gentoo on Nexus 6P <a href="https://jsteward.moe/booting-gentoo-on-nexus-6p.html">in the previous article</a>, we can move on to launching Android in LXC, which is the last mission in the first period of my GSoC 2018 project. This article documents the process to bring up Android successfully with most of its functions …</p><h2>Preface</h2>
<p>After successfully booting Gentoo on Nexus 6P <a href="https://jsteward.moe/booting-gentoo-on-nexus-6p.html">in the previous article</a>, we can move on to launching Android in LXC, which is the last mission in the first period of my GSoC 2018 project. This article documents the process to bring up Android successfully with most of its functions working; only a few things don't work by now, and they're recorded in the Known Problems section at the end of this article. Note that while this article strives to provide easy-to-follow instructions, it may not be suitable as a step-by-step tutorial. That is to say, you may encounter problems due to mistakes or things that the author didn't notice. The author appreciates your understanding when such things happen; meanwhile, feel free to contact the author by the means listed in the page linked at the bottom of this page.</p>
<h2>Legends</h2>
<p>Command prompts in this article starts with the hostname of the machine, followed by a <code>#</code> or <code>$</code>, denoting if the user executing the command need to be root or not. The following hostnames have special meanings:</p>
<ul>
<li><code>yuki</code>: the Linux workstation</li>
<li><code>umi</code>: Gentoo Linux on Nexus 6P</li>
<li><code>angler</code>: Android on Nexus 6P</li>
</ul>
<h2>Set Up Filesystem for Android</h2>
<p>Android directly uses the initramfs as its rootfs and mounts other partitions, namely <code>userdata</code>, <code>system</code>, <code>cache</code>, <code>persist</code>, and <code>vendor</code>, at mountpoints in <code>/</code>. What we need to do here is to copy Android's rootfs to LXC's rootfs location so that we can launch it later on. Grab a copy of the Android <code>boot.img</code> (not the trimmed <code>preinit</code> <code>boot.img</code>), unpack it with the <a href="https://github.com/KireinaHoro/preinit_angler/blob/61f8fefb1f027eb09dc6ee291f6f990cf032c624/Makefile#L64">Makefile rules in <code>preinit</code> repository</a>, tar it up, and transfer the tarball to the phone via adb. Remember to enable root access for ADB, which can be found in Developer Options in Android.</p>
<div class="highlight"><pre><span></span><code><span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">adb</span><span class="w"> </span><span class="nv">devices</span>
<span class="nv">List</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">devices</span><span class="w"> </span><span class="nv">attached</span>
<span class="mi">84</span><span class="nv">xxxxxxxxxxxxxx</span><span class="w"> </span><span class="nv">device</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">adb</span><span class="w"> </span><span class="nv">root</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">adb</span><span class="w"> </span><span class="nv">shell</span><span class="w"> </span><span class="nv">dd</span><span class="w"> </span><span class="k">if</span><span class="o">=/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">block</span><span class="o">/</span><span class="nv">bootdevice</span><span class="o">/</span><span class="nv">by</span><span class="o">-</span><span class="nv">name</span><span class="o">/</span><span class="nv">boot</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">dd</span><span class="w"> </span><span class="nv">of</span><span class="o">=</span><span class="nv">boot</span><span class="o">-</span><span class="nv">stock</span><span class="o">.</span><span class="nf">img</span>
<span class="p">(</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nv">output</span><span class="w"> </span><span class="nv">elided</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">)</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">mv</span><span class="w"> </span><span class="nv">boot</span><span class="o">-</span><span class="nv">stock</span><span class="o">.</span><span class="nv">img</span><span class="w"> </span>~<span class="o">/</span><span class="nv">preinit_angler</span><span class="o">/</span><span class="nv">initramfs</span><span class="o">/</span><span class="nv">boot</span><span class="o">.</span><span class="nv">img</span><span class="w"> </span>&&<span class="w"> </span><span class="nv">cd</span><span class="w"> </span>~<span class="o">/</span><span class="nv">preinit_angler</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">make</span><span class="w"> </span><span class="nf">unpack</span>
<span class="p">(</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nv">output</span><span class="w"> </span><span class="nv">elided</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">)</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">cd</span><span class="w"> </span><span class="nv">initramfs</span><span class="o">/</span><span class="nv">root</span><span class="w"> </span>&&<span class="w"> </span><span class="nv">tar</span><span class="w"> </span><span class="nv">czvf</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">.</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">adb</span><span class="w"> </span><span class="nv">shell</span><span class="w"> </span><span class="s">"cat >/data/android-root.tar.gz"</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span>
</code></pre></div>
<p>Copy the tarball into the Gentoo chroot and enter the chroot. Create the necessary mountpoints and unpack the root tarball into LXC rootfs.</p>
<div class="highlight"><pre><span></span><code><span class="n">yuki</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">adb</span><span class="w"> </span><span class="n">shell</span>
<span class="n">angler</span><span class="w"> </span><span class="c1"># /data/gnu/start_chroot</span>
<span class="n">angler</span><span class="w"> </span><span class="c1"># mv /data/android-root.tar.gz /data/gnu/root/</span>
<span class="n">angler</span><span class="w"> </span><span class="c1"># chroot /data/gnu /bin/su</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># mkdir -p /var/lib/android/{system,vendor,data,system,cache}</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># mkdir -p /var/lib/lxc/android/rootfs</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># tar xvf ~/android-root.tar.gz -C /var/lib/lxc/android/rootfs</span>
<span class="p">(</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="n">elided</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">)</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># mkdir -p /var/lib/lxc/android/run</span>
<span class="n">umi</span><span class="w"> </span><span class="c1">#</span>
</code></pre></div>
<p>We need <code>/run</code> in Android rootfs to talk to the host for the <code>charger</code> hack later on.</p>
<p>Write the following <code>fstab</code> entries to <code>/etc/fstab</code> in Gentoo, so that OpenRC will take care of mounts on Gentoo startup.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Android mounts</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mmcblk0p43</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">system</span><span class="w"> </span><span class="n">ext4</span><span class="w"> </span><span class="n">ro</span><span class="p">,</span><span class="n">barrier</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="n">inode_readahead_blks</span><span class="o">=</span><span class="mi">8</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mmcblk0p32</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">persist</span><span class="w"> </span><span class="n">ext4</span><span class="w"> </span><span class="n">noatime</span><span class="p">,</span><span class="n">nosuid</span><span class="p">,</span><span class="n">nodev</span><span class="p">,</span><span class="n">barrier</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="n">data</span><span class="o">=</span><span class="n">ordered</span><span class="p">,</span><span class="n">nomblk_io_submit</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mmcblk0p37</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">vendor</span><span class="w"> </span><span class="n">ext4</span><span class="w"> </span><span class="n">ro</span><span class="p">,</span><span class="n">barrier</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="n">inode_readahead_blks</span><span class="o">=</span><span class="mi">8</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mmcblk0p38</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">cache</span><span class="w"> </span><span class="n">ext4</span><span class="w"> </span><span class="n">noatime</span><span class="p">,</span><span class="n">nosuid</span><span class="p">,</span><span class="n">nodev</span><span class="p">,</span><span class="n">barrier</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="n">data</span><span class="o">=</span><span class="n">ordered</span><span class="p">,</span><span class="n">nomblk_io_submit</span><span class="p">,</span><span class="n">noauto_da_alloc</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mmcblk0p44</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">data</span><span class="w"> </span><span class="n">ext4</span><span class="w"> </span><span class="n">noatime</span><span class="p">,</span><span class="n">nosuid</span><span class="p">,</span><span class="n">nodev</span><span class="p">,</span><span class="n">barrier</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="n">data</span><span class="o">=</span><span class="n">ordered</span><span class="p">,</span><span class="n">nomblk_io_submit</span><span class="p">,</span><span class="n">noauto_da_alloc</span><span class="p">,</span><span class="n">inode_readahead_blks</span><span class="o">=</span><span class="mi">8</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="c1"># Bind into container</span>
<span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">system</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">system</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">persist</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">persist</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">vendor</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">vendor</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">cache</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">cache</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">data</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">data</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">run</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">lxc</span><span class="o">/</span><span class="n">android</span><span class="o">/</span><span class="n">rootfs</span><span class="o">/</span><span class="n">run</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>Boot into Gentoo by temporarily booting the <code>preinit</code> <code>boot.img</code> to check if the partitions are mounted correctly. Double-check that the device nodes are correct if things go wrong. Boot back into Android by rebooting.</p>
<h2>Install LXC in Gentoo Linux</h2>
<p>LXC is the core suite to boot Android containerized here. The installation process is in two parts: installing the userspace utilities, and enabling necessary options in the kernel.</p>
<h3>Userspace Utilities</h3>
<p>This is not different from installing normal Gentoo packages. Enter the chroot and install <code>app-emulation/lxc</code>:</p>
<div class="highlight"><pre><span></span><code>yuki $ adb root
yuki $ adb shell /data/gnu/start_chroot
yuki $ adb shell chroot /data/gnu /bin/su
umi # emerge --verbose --autounmask-write=y app-emulation/lxc
( ... output elided ... )
umi # dispatch-conf
( ... output elided ... )
umi # emerge --verbose app-emulation/lxc
( ... output elided ... )
umi #
</code></pre></div>
<p>The process can take a little bit of time to run. If it's too slow in your case, consider <a href="https://wiki.gentoo.org/wiki/Cross_build_environment">setting up a cross build environment</a> per instructions on Gentoo Wiki if you have a Gentoo workstation. That's out of the scope of this article.</p>
<h3>Kernel</h3>
<p>According to <a href="https://wiki.gentoo.org/wiki/LXC#Kernel_with_the_appropriate_LXC_options_enabled">LXC on Gentoo Wiki</a>, we need quite many options for LXC's functioning. I've put how to compile a custom kernel in a separate article: <a href="https://jsteward.moe/building-lxc-ready-kernel.html">Building a LXC-ready Android kernel with Gentoo toolchain</a>. Follow that article to get a <code>Image.gz-dtb</code> file, which is the kernel image that satisfies our needs, and proceed with the following instructions to repack the <code>preinit</code> <code>boot.img</code>--we no longer need the Android <code>boot.img</code> to be <code>boot</code> from this point on.</p>
<div class="highlight"><pre><span></span><code><span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">cp</span><span class="w"> </span><span class="nv">Image</span><span class="o">.</span><span class="nv">gz</span><span class="o">-</span><span class="nv">dtb</span><span class="w"> </span>~<span class="o">/</span><span class="nv">preinit_angler</span><span class="o">/</span><span class="nv">out</span><span class="o">/</span><span class="nv">zImage</span><span class="w"> </span>&&<span class="w"> </span><span class="nv">cd</span><span class="w"> </span>~<span class="o">/</span><span class="nv">preinit_angler</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">make</span><span class="w"> </span><span class="nf">repack</span>
<span class="p">(</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nv">output</span><span class="w"> </span><span class="nv">elided</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">)</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span>
</code></pre></div>
<p>The final <code>preinit</code> <code>boot.img</code> should be located at <code>out/boot-mod.img</code>. Boot the phone into <code>fastboot</code>, then <em>flash</em> the new image.</p>
<div class="highlight"><pre><span></span><code><span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">fastboot</span><span class="w"> </span><span class="nv">devices</span>
<span class="mi">84</span><span class="nv">xxxxxxxxxxxxxx</span><span class="w"> </span><span class="nv">fastboot</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span><span class="w"> </span><span class="nv">fastboot</span><span class="w"> </span><span class="nv">flash</span><span class="w"> </span><span class="nv">boot</span><span class="w"> </span><span class="nv">out</span><span class="o">/</span><span class="nv">boot</span><span class="o">-</span><span class="nv">mod</span><span class="o">.</span><span class="nf">img</span>
<span class="p">(</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nv">output</span><span class="w"> </span><span class="nv">elided</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">)</span>
<span class="nv">yuki</span><span class="w"> </span><span class="p">$</span>
</code></pre></div>
<p>Boot into Gentoo by choosing "Start" in <code>fastboot</code>. Run <code>lxc-checkconfig</code> to see if our kernel is suitable. Not everything's required; as long as you don't see red "missing" on items other than systemd, you're good to go.</p>
<div class="highlight"><pre><span></span><code><span class="n">umi</span><span class="w"> </span><span class="p">#</span><span class="w"> </span><span class="n">lxc</span><span class="o">-</span><span class="n">checkconfig</span>
<span class="o">---</span><span class="w"> </span><span class="n">Namespaces</span><span class="w"> </span><span class="o">---</span>
<span class="n">Namespaces</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Utsname</span><span class="w"> </span><span class="n">namespace</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Ipc</span><span class="w"> </span><span class="n">namespace</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Pid</span><span class="w"> </span><span class="n">namespace</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">User</span><span class="w"> </span><span class="n">namespace</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Network</span><span class="w"> </span><span class="n">namespace</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Multiple</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">pts</span><span class="w"> </span><span class="n">instances</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="o">---</span><span class="w"> </span><span class="n">Control</span><span class="w"> </span><span class="n">groups</span><span class="w"> </span><span class="o">---</span>
<span class="n">Cgroups</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">v1</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">points</span><span class="p">:</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">openrc</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">cpuset</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">debug</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">cpu</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">cpuacct</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">devices</span>
<span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">cgroup</span><span class="o">/</span><span class="n">freezer</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">v2</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">points</span><span class="p">:</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">v1</span><span class="w"> </span><span class="n">systemd</span><span class="w"> </span><span class="n">controller</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">v1</span><span class="w"> </span><span class="n">clone_children</span><span class="w"> </span><span class="n">flag</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">device</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">sched</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">cpu</span><span class="w"> </span><span class="n">account</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">memory</span><span class="w"> </span><span class="n">controller</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">Cgroup</span><span class="w"> </span><span class="n">cpuset</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="o">---</span><span class="w"> </span><span class="n">Misc</span><span class="w"> </span><span class="o">---</span>
<span class="n">Veth</span><span class="w"> </span><span class="n">pair</span><span class="w"> </span><span class="n">device</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">Macvlan</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">Vlan</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">Bridges</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">Advanced</span><span class="w"> </span><span class="n">netfilter</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">CONFIG_NF_NAT_IPV4</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">CONFIG_NF_NAT_IPV6</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">CONFIG_IP_NF_TARGET_MASQUERADE</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">CONFIG_IP6_NF_TARGET_MASQUERADE</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">CONFIG_NETFILTER_XT_TARGET_CHECKSUM</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">CONFIG_NETFILTER_XT_MATCH_COMMENT</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="n">FUSE</span><span class="w"> </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">lxcfs</span><span class="p">):</span><span class="w"> </span><span class="n">enabled</span><span class="p">,</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">loaded</span>
<span class="o">---</span><span class="w"> </span><span class="n">Checkpoint</span><span class="o">/</span><span class="n">Restore</span><span class="w"> </span><span class="o">---</span>
<span class="n">checkpoint</span><span class="w"> </span><span class="n">restore</span><span class="p">:</span><span class="w"> </span><span class="n">missing</span>
<span class="n">CONFIG_FHANDLE</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">CONFIG_EVENTFD</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">CONFIG_EPOLL</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">CONFIG_UNIX_DIAG</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">CONFIG_INET_DIAG</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">CONFIG_PACKET_DIAG</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">CONFIG_NETLINK_DIAG</span><span class="p">:</span><span class="w"> </span><span class="n">enabled</span>
<span class="n">File</span><span class="w"> </span><span class="n">capabilities</span><span class="p">:</span>
<span class="n">Note</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">Before</span><span class="w"> </span><span class="n">booting</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">kernel</span><span class="p">,</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">check</span><span class="w"> </span><span class="n">its</span><span class="w"> </span><span class="n">configuration</span>
<span class="n">usage</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">CONFIG</span><span class="o">=/</span><span class="n">path</span><span class="o">/</span><span class="k">to</span><span class="o">/</span><span class="n">config</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">lxc</span><span class="o">-</span><span class="n">checkconfig</span>
<span class="n">umi</span><span class="w"> </span><span class="p">#</span>
</code></pre></div>
<h2>Bring Android up in LXC container</h2>
<p>Now that we have LXC correctly configured, we can set up the Android container. Some modifications to Andoid's <code>fstab.<device></code> and <code>init.rc</code> are needed for a successful boot. Some hacks are required to fix some functions, of which we'll discuss in the next section. Commands in this section are expected to be executed with Gentoo booted up with <code>preinit</code>, not in the chroot from Android.</p>
<p>The modified files' patches as well as helper scripts are put together in <a href="https://github.com/KireinaHoro/android-lxc-files">a repository on GitHub</a>. Make sure that you apply them before continuing.</p>
<p>Place <code>pre-start.sh</code>, <code>post-stop.sh</code>, and <code>config</code> in <code>/var/lib/lxc/android</code>. Make sure that the executable bit is there for the scripts. Issue the following to see if LXC has recognized the container:</p>
<div class="highlight"><pre><span></span><code>umi # lxc-info android
Name: android
State: STOPPED
</code></pre></div>
<p>If everything's going as expected, start the container, and we'll see Android booting up on the device screen.</p>
<div class="highlight"><pre><span></span><code>umi # lxc-start android
umi #
</code></pre></div>
<p>If the container failed to start, issue the following and then examine <code>start.log</code> to see what's wrong:</p>
<div class="highlight"><pre><span></span><code>umi # lxc-start -l debug -o start.log -n android
( ... error output elided ... )
umi #
</code></pre></div>
<p>Verify that things are working, especially the hardware. In case something's crashing, or the Android framework failed to start at all, connect the phone to the computer via USB and check out <code>logcat</code> via ADB (it starts pretty early). Tombstones at <code>/data/tombstones</code> in Android root are usually helpful; consult <a href="https://source.android.com/devices/tech/debug/">this article</a> for how to read them.</p>
<div class="highlight"><pre><span></span><code>yuki $ adb logcat | less
yuki $ adb shell
angler # cd /data/tombstones
</code></pre></div>
<p>Only some small bits don't work by now, and we'll cover them in the following section.</p>
<h2>Improvements</h2>
<h3>Container Auto-start</h3>
<p>It's tiresome to attach a serial cable and issue <code>lxc-start android</code> to boot Android every time. Create <code>lxc.android</code> service and add it to the <code>boot</code> runlevel, as Android takes care of most of the device-specific work, namely networking:</p>
<div class="highlight"><pre><span></span><code>umi # cd /etc/init.d
umi # ln -s lxc lxc.android
umi # rc-update add lxc.android boot
</code></pre></div>
<h3>Dial Home from Android world</h3>
<p>Having to attach serial cable to access Gentoo is just too tiresome. Configure <code>ssh</code> so that the Android guest can <code>dialhome</code>.</p>
<div class="highlight"><pre><span></span><code>angler # ssh-keygen -t ed25519 -C android -f /data/ssh/id_ed25519
</code></pre></div>
<p>Enable <code>sshd</code> in Gentoo, configure <code>authorized_keys</code> for root, and put a little script <code>dialhome</code> (available in <a href="https://github.com/KireinaHoro/android-lxc-files/tree/master/scripts"><code>scripts/</code> in the <code>android-lxc-files</code> repository</a>) for quick access:</p>
<div class="highlight"><pre><span></span><code><span class="n">umi</span><span class="w"> </span><span class="c1"># umask 077</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># mkdir -p ~/.ssh</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># cat /var/lib/android/data/ssh/id_ed25519.pub >> ~/.ssh/authorized_keys</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># cp dialhome /var/lib/android/data/ssh/</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># chmod u+x /var/lib/android/data/ssh/dialhome</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># chown 2000:2000 /var/lib/android/data/ssh/dialhome</span>
<span class="n">umi</span><span class="w"> </span><span class="c1"># rc-update add sshd default</span>
</code></pre></div>
<p>We can now issue <code>/data/ssh/dialhome</code> to log into Gentoo. No more serial cable needed unless something goes wrong. We can safely disable ADB root access from Android Developer Settings by now.</p>
<p>Note that you can <code>lxc-attach android</code> from Gentoo world! This will launch <code>/bin/sh</code>, which is the busybox sh. You may need to set <code>PATH</code> to include <code>/system/bin</code> to get Android's utilities.</p>
<h3>Fix Charger Mode</h3>
<p>Android's init knows whether it's plugged into the charger while powered off. If that's the case, it launches <code>/sbin/charger</code>, which draws the charging animation on the screen. Long-pressing the power button causes a restart, and the system will boot into normal Android. Unfortunately, our LXC stop hooks can't tell if the container ended with a poweroff or a restart, thus we won't be able to exit charger mode unless we unplug the power source.</p>
<p>The solution is quite simple: replace <code>/sbin/charger</code> with a simple script that places a marker file in <code>/run</code> (which is also GNU/Linux's <code>/run</code>), and then call the real charger, which is now <code>/sbin/charger.real</code>. The <code>post-stop.sh</code> hook can then reboot if it sees that marker file. The script is available <a href="https://github.com/KireinaHoro/android-lxc-files/blob/master/scripts/charger">here</a>.</p>
<h2>How It Worked</h2>
<p>If you take a look at the config for LXC, the start-stop hooks, and the <code>init.rc</code> patch, you'll discover that we needed to correct paths for cgroup pseudo-filesystems and set <code>rt_runtime_us</code> RT schedule parameters. Android uses cgroups heavily as well, and we need to get it correct so that Android doesn't end up writing to the host's cgroup space. We need to set <code>rt_runtime_us</code> to allocate realtime bandwidth to the cgroup, otherwise realtime schedulers will fail. The following should be helpful to grab a general understanding of group scheduling:</p>
<ul>
<li><a href="https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt">Real-Time group scheduling</a></li>
<li><a href="https://serverfault.com/questions/630220/how-do-i-configure-lxc-to-allow-the-use-of-sched-rr-in-a-container/632564">How do I configure LXC to allow the use of SCHED_RR in a container?</a></li>
</ul>
<p>The rest of things (filesystem mounting, etc.) are pretty straightforward. Note that we need to tell Android (<code>vold</code> to be specific) not to touch mounting, by commenting out <code>fstab</code> entries for the relevant partitions. Due to <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp#2919">poorly-written <code>vold</code> code without <code>nullptr</code> checks</a>, we need an empty entry for <code>/data</code> in Android <code>fstab</code>, otherwise <code>vold</code> will crash.</p>
<p>We bind <code>/run</code> from GNU/Linux Android so that the container can talk to the host, informing the host about its states (such as "We're in charger mode: reboot when I finish instead of power off"). <code>/run</code> was chosen because it's the place to hold runtime files by convention on a GNU/Linux system. Read <a href="http://www.h-online.com/open/news/item/Linux-distributions-to-include-run-directory-1219006.html">this article</a> for more information.</p>
<h2>Known Problems</h2>
<p>The (non-exhaustive) list of problems as of 2018-05-26:</p>
<ul>
<li>Reboot function in Android system (along with Advanced Reboot) doesn't work. This is because Android system doesn't actually poweroff the system or reboot it when inside a container: it simply performs <code>exit(0)</code>, making it impossible for the GNU/Linux world (at least for now) to tell how the container stopped. The current behavior is that the system will power down regardless of whether "Poweroff" or "Reboot" (or other reboot options, e.g. Reboot to bootloader) has been selected in Android.<ul>
<li><code>charger</code> suffers from similar problems, though a workaround works pretty well; see The Charger section above.</li>
<li>This may be fixed by patching Android framework so that it signals the host about how it ended, by placing a file in the <code>/run</code> filesystem bind-mounted from the host. The host then decides whether to reboot or to poweroff according to the signal file. This is exactly how <code>charger</code>'s problem gets fixed.</li>
</ul>
</li>
<li>Boot is unbearably slow. Charging while powered off takes a long time to show the charging animation as well.<ul>
<li>This is because the <em>Linux kernel's initialization time</em> is long. It takes over 10 seconds for the kernel to launch the first userspace program--<code>preinit</code>.</li>
<li>For better experience, the kernel should have most of its functions as kernel modules, so that we can display a boot animation (or a static picture other than the Google logo from bootloader) as early as possible. We may draw an animation that's unrelated to <code>bootanim</code> in Android though.</li>
</ul>
</li>
<li>Full-disk encryption / file-based encryption doesn't work.<ul>
<li>This was done deliberately, as Android's <code>cryptfs</code> implementation is a little bit too complicated to implement (see <a href="https://jsteward.moe/analysis-of-android-cryptfs.html">my previous analysis on Android cryptfs</a>).</li>
<li>Standard encryption means should get implemented at some point (e.g. LUKS).</li>
<li>Choosing to encrypt device in System Settings in Android can cause <strong>unexpected behaviors</strong>. <em>YOU HAVE BEEN WARNED.</em></li>
</ul>
</li>
</ul>Buidling a LXC-ready Android kernel with Gentoo toolchain2018-05-26T10:00:00+08:002018-05-26T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-26:/building-lxc-ready-kernel.html<h2>Preface</h2>
<p>To run LXC, we need a kernel that has the options required by LXC on. This article shows how to build an Android kernel that has the required options on, built by Gentoo's latest stable cross-compile toolchain built by <code>crossdev</code>.</p>
<h2>Install Cross-Compile Toolchain</h2>
<p>We'll use <a href="https://wiki.gentoo.org/wiki/Crossdev">crossdev</a>, a set of …</p><h2>Preface</h2>
<p>To run LXC, we need a kernel that has the options required by LXC on. This article shows how to build an Android kernel that has the required options on, built by Gentoo's latest stable cross-compile toolchain built by <code>crossdev</code>.</p>
<h2>Install Cross-Compile Toolchain</h2>
<p>We'll use <a href="https://wiki.gentoo.org/wiki/Crossdev">crossdev</a>, a set of wrapper scripts that provides cross-compilation capability to Portage. The following instructions assume that the workstation is a Gentoo machine.</p>
<p>First, install <code>sys-devel/crossdev</code>:</p>
<div class="highlight"><pre><span></span><code>emerge -av sys-devel/crossdev
</code></pre></div>
<p>Install the latest stable toolchain:</p>
<div class="highlight"><pre><span></span><code>crossdev --stable -t aarch64-unknown-linux-gnu
</code></pre></div>
<h2>Clone the kernel sources and compile</h2>
<p>The kernel source repository for Nexus 6P with my patches for compiling with GCC 6 as well as LXC-enabled config is at <a href="https://github.com/KireinaHoro/android_kernel_huawei_angler"><code>KireinaHoro/android_kernel_huawei_angler</code></a>. Clone this repository and compile the kernel:</p>
<div class="highlight"><pre><span></span><code>git clone https://github.com/KireinaHoro/android_kernel_huawei_angler
cd android_kernel_huawei_angler
make sharkbait_angler_defconfig
make -j8
</code></pre></div>
<p>The target kernel image will be available at <code>arch/arm64/Image.gz-dtb</code>. Copy this to the <code>preinit</code> repository to continue following <a href="https://jsteward.moe/starting-android-in-lxc.html">Starting Android in LXC</a> to build the <code>preinit</code> <code>boot.img</code>.</p>
<h2>Notes</h2>
<p>Linux kernel's <code>make savedefconfig</code> doesn't really save a working <code>defconfig</code>, at least it's not the case for Android kernels. When changes are made to <code>.config</code> (via <code>make menuconfig</code> or so), <strong>copy <code>.config</code> back to <code>arch/arm64/configs</code></strong> instead of relying on the <code>savedefconfig</code> target, otherwise you may encounter problems such as missing drivers or things that should be built-in turned out to be built as a module.</p>
<p>GCC 6 support was accomplished by introducing <a href="https://gitlab.com/SaberMod/android-kernel-lge-hammerhead/blob/4a98feebe2cb735eef2299bc5512b9c83cc789ce/include/linux/compiler-gcc6.h"><code>compiler-gcc6.h</code> from SaberMod/android-kernel-lge-hammerhead on GitLab</a>. A few fixes regarding the <code>static inline</code> behavior change since GCC 6 has been made to make the source compile without problem. Warning checks in <code>gcc-wrapper.py</code> was disabled as well.</p>Booting Gentoo on Nexus 6P2018-05-25T20:00:00+08:002018-05-25T20:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-25:/booting-gentoo-on-nexus-6p.html<h2>Preface</h2>
<p>As we now have <code>preinit</code> and the crucial UART console available, we can start bringing up the real GNU/Linux system--Gentoo Linux in this case. This article will focus on the following topics:</p>
<ul>
<li>Filesystem structure and mounting procedure (in details)</li>
<li>Launch sequence to bring up OpenRC</li>
<li>Crafting a <code>preinit …</code></li></ul><h2>Preface</h2>
<p>As we now have <code>preinit</code> and the crucial UART console available, we can start bringing up the real GNU/Linux system--Gentoo Linux in this case. This article will focus on the following topics:</p>
<ul>
<li>Filesystem structure and mounting procedure (in details)</li>
<li>Launch sequence to bring up OpenRC</li>
<li>Crafting a <code>preinit</code> initramfs</li>
<li>Tweaks in Gentoo to get ready to start Android in LXC later on</li>
</ul>
<h2>Filesystem Structure and Mounting Procedure</h2>
<p>We won't be changing the partition table, as altering that will prevent <code>fastboot flash</code> commands from working properly, and we won't be able to recover unless we re-flash the EMMC chip if we messed the partition table up. The partition we'll use for GNU/Linux root is <code>userdata</code>, but not the entire filesystem--we can retain compatibility with stock Android <code>boot.img</code> by preserving <code>userdata</code> as the partition to be mounted for Android <code>/data</code>. The Gentoo root filesystem sits at <code>userdata:/gnu</code>, and gets binded to <code>/</code> in <code>preinit</code>.</p>
<p>After OpenRC boots, <code>userdata</code>, <code>system</code>, <code>vendor</code>, <code>persist</code>, and <code>cache</code> gets mounted to their mountpoints under <code>/var/lib/android</code> for launching Android later on. This is done by the <code>localmount</code> service in OpenRC's <code>sysinit</code> runlevel, by reading <code>/etc/fstab</code> in Gentoo root. These partitions later get bind-mounted into Android's LXC rootfs, which will be the topic for my next article.</p>
<h2>Launch Sequence to Bring up OpenRC</h2>
<p>Once kernel finishes its early initializations, <code>preinit</code>, a busybox sh script, gets called. The source is available <a href="https://github.com/KireinaHoro/preinit_angler/blob/141cccc95e0f925e2e1be3cfd02d1e12eabef577/src/init">here in the <code>preinit</code> repository</a>. Preinit basically does the following:</p>
<ul>
<li>Mount pseudo-filesystems for <code>preinit</code>'s proper functioning</li>
<li>Mount <code>userdata</code> partition at <code>/mnt/userdata</code></li>
<li>Bind-mount <code>/mnt/userdata/gnu</code> to itself, making it a mountpoint, which was mandatory for <code>switch_root</code></li>
<li><code>exec switch_root</code> into <code>/mnt/userdata/gnu</code> and call <code>/sbin/init</code> </li>
</ul>
<p>This procedure does not differ much from standard <code>initramfs</code>es, except that the desired root is not the filesystem root. We can add other things into <code>preinit</code> later on to support alternative root layouts, e.g. LUKS.</p>
<h2>Crafting a Preinit initramfs</h2>
<p>As we're not launching anything other than <code>/init</code>, and, unlike Android, we have a real root filesystem instead of the entire root on <code>tmpfs</code>, we can safely trim unnecessary components inside the initramfs. What we really need in the initramfs is as follows:</p>
<ul>
<li><code>init</code>: busybox sh script (duh)</li>
<li><code>busybox</code>: static executable</li>
<li>symlinks to <code>busybox</code> applets under <code>/bin</code></li>
<li>mountpoints</li>
</ul>
<p>The complete <code>initramfs</code> layout can be found in <a href="https://github.com/KireinaHoro/preinit_angler/tree/141cccc95e0f925e2e1be3cfd02d1e12eabef577/initramfs/root">the <code>preinit</code> repository</a>. Repack <code>boot.img</code> with the Makefile rules at the root of <code>preinit</code> repository. Remember to turn SELinux to <code>permissive</code> in kernel commandline: though OpenRC won't respect that commandline option, Android init will load the policies and enforce it later on, and as we're not SELinux-ready now, that would certainly break things.</p>
<p>Boot the phone into <code>fastboot</code> mode, but don't hurry to flash that <code>boot.img</code> as the <code>boot</code> partition yet--use the following commandline to boot the new <code>boot.img</code> temporarily, so that we still have the regular Android <code>boot.img</code> for debug and further actions that require a working Internet connection:</p>
<div class="highlight"><pre><span></span><code>fastboot boot <modded boot.img>
</code></pre></div>
<p>Attach the UART cable to see what's going on. If the kernel can't launch init, double-check that the <code>init</code> script is executable, and the shebang line is correct. Also check if the symlinks to <code>busybox</code> are placed correctly. If everything goes as expected, you should see our dear Preinit and OpenRC waving hello to you:</p>
<div class="highlight"><pre><span></span><code>[ 12.602544] new_era: Preinit started
mount: /etc/mtab: No such file or directory
mount: /etc/mtab:[ 12.606728] new_era: Trying to mount /dev/mmcblk0p44 on /mnt/userdata...
No such file or directory
[ 12.735390] EXT4-fs (mmcblk0p44): mounted filesystem with ordered data mode. Opts: (null)
mount: /etc/mtab: No such file or directory
[ 12.743008] new_era: Setting up /mnt/userdata/gnu as mountpoint...
mount: /etc/mtab: No such file or directory
[ 12.767932] new_era: Cleaning up mounts, switching root to /mnt/userdata/gnu, and launching /sbin/init...
mount: /etc/mtab: No such file or directory
mount: /etc/mtab: No such file or directory
mount: /etc/mtab: No such file or directory
INIT: version 2.88 booting
OpenRC 0.34.11 is starting up Gentoo Linux (aarch64)
<span class="k">*</span> /proc is already mounted
<span class="k">*</span> Mounting /run ...
<span class="k">*</span> /run/openrc: creating directory
<span class="k">*</span> /run/lock: creating directory
<span class="k">*</span> /run/lock: correcting owner
<span class="k">*</span> Caching service dependencies ...
</code></pre></div>
<h2>Tweaks in Gentoo to get ready to start Android in LXC later on</h2>
<p>Though the system is up and running, we need to tweak a few things to make it suit our later launching Android in LXC. In case a shell is not yet available on serial console (which is most likely the case by default), boot into Android and <code>chroot</code> into Gentoo root as described in <a href="/building-gentoo-chroot-in-android.html">this article</a> to configure the following.</p>
<p>You'll notice that we don't have a <code>getty</code> on the serial console: <code>sysvinit</code> does not start <code>getty</code> on <code>ttyHSL0</code> by default. Edit <code>/etc/inittab</code> in Gentoo root, comment out <code>tty*</code> lines (which Android devices definitely don't have) and add the following line:</p>
<div class="highlight"><pre><span></span><code><span class="n">s0</span><span class="o">:</span><span class="mi">12345</span><span class="o">:</span><span class="n">respawn</span><span class="o">:/</span><span class="n">sbin</span><span class="o">/</span><span class="n">agetty</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="mi">115200</span><span class="w"> </span><span class="n">ttyHSL0</span><span class="w"> </span><span class="n">vt100</span>
</code></pre></div>
<p>Remove <code>udev</code> from <code>sysinit</code> runlevel. It inteferes with Android's <code>ueventd</code>, and sometimes it takes ages for <code>udev</code> to process the <code>uevents</code> from the kernel. <code>keymaps</code> and <code>termencoding</code> are useless as well, as we'll only use a serial console, on which these services don't make sense.</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span>{<span class="nv">udev</span>,<span class="nv">keymaps</span>,<span class="nv">termencoding</span>}<span class="c1">; do rc-update del $a boot; done</span>
</code></pre></div>
<p>Though we do not have networking by now, add <code>sshd</code> to the <code>default</code> runlevel. Add <code>syslog-ng</code> as well.</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span>{<span class="nv">sshd</span>,<span class="nv">syslog</span><span class="o">-</span><span class="nv">ng</span>}<span class="c1">; do rc-update add $a default; done</span>
</code></pre></div>
<p>Reboot. The system should now be running properly. In the next article, we'll boot back into Android to install LXC in Gentoo root, and we'll then launch Android in LXC.</p>UART on Nexus 6P2018-05-22T13:00:00+08:002018-05-22T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-22:/nexus-6p-uart.html<h2>The story</h2>
<p>As described <a href="https://android.googlesource.com/device/google/debugcable">here</a>, Google baked a serial console into the headphone jack of their Nexus and Pixel devices. A serial console would be feasible for debugging problems with <code>init</code> as well as tinkering the device when it has booted into Linux and haven't started Android yet. My journey …</p><h2>The story</h2>
<p>As described <a href="https://android.googlesource.com/device/google/debugcable">here</a>, Google baked a serial console into the headphone jack of their Nexus and Pixel devices. A serial console would be feasible for debugging problems with <code>init</code> as well as tinkering the device when it has booted into Linux and haven't started Android yet. My journey to successfully build such a cable turned out to be quite complicated, though.</p>
<h2>PL2303</h2>
<p>PL2303 is a chip that provides 5v and 3v3 TTL tx/rx, which serves as a serial port to communicate with the phone over UART. @imi415 on Telegram made a cable with circuits to adjust the voltages to match what the phone accepts (1.8v); however, it's not working properly: the bootloader log reads just fine, yet the kernel message won't get to the computer.</p>
<p>After visiting @imi415's lab and having the output voltages from the phone measured, we discovered that the bootloader made output at 2.5v, while the kernel made output at 1.9v. As PL2303 can't properly handle 1.8v rx, this resulted in the computer not receiving anything at all after kernel starts.</p>
<h2>FT232R</h2>
<p>@imi415 suggested that I should try FT232, as it supported 1.8v input; and <a href="http://people.redhat.com/jmcnicol/nexus_debug/">this post</a> showed that chips from FTDI at 3.3v mode should work fine on Nexus 5X. As Nexus 5X shouldn't be different from Nexus 6P too much, I purchased a FT232RL TTL board and soldered the headphone cable to some jump wires, and connected them to the FT232RL unit. Plugging the unit into the computer and firing up PuTTY proved that things worked perfectly. Credit for the solder work goes to @Catofes on Telegram.</p>
<h2>Pictures</h2>
<p>FT232RL TTL Board:</p>
<p><img alt="FT232RL TTL Board" src="/images/ft232rl.jpg"></p>
<p>FT232RL with voltage converter (but didn't work :/):</p>
<p><img alt="FT232RL with voltage converter (but didn't work :/)" src="/images/ft232rl-with-voltage-converter.jpg"></p>
<p>Bypassing the voltage converter:</p>
<p><img alt="Bypassing the voltage converter" src="/images/bypassing-voltage-converter.jpg"></p>
<p>Before soldering the wires:</p>
<p><img alt="Before soldering the wires" src="/images/before-soldering-wires-together.jpg"></p>
<p>Final product:</p>
<p><img alt="Final product" src="/images/final-product.jpg"></p>
<p>Console output!</p>
<p><img alt="Console output!" src="/images/console-output.jpg"></p>Gentoo GSoC Weekly Report 05/212018-05-21T19:00:00+08:002018-05-21T19:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-21:/weekly-report-0521.html<h2>Things done this week</h2>
<p>Due to having to take an exam in Algebra last Friday, I revised for the test this week and didn't have much progress in GSoC this week. This article sums up what I've tried to achieve in the past week.</p>
<h2>Makefile to unpack and repack boot …</h2><h2>Things done this week</h2>
<p>Due to having to take an exam in Algebra last Friday, I revised for the test this week and didn't have much progress in GSoC this week. This article sums up what I've tried to achieve in the past week.</p>
<h2>Makefile to unpack and repack boot.img without AOSP source tree</h2>
<p>As I'm starting to hack on Android's <code>boot.img</code>, I'd be frequently unpacking and repacking it, making the old way of patching the initramfs during the AOSP build very inconvenient (a simple repack would require sourcing Android Build System's thousands of Makefiles). I created rules for unpacking and repacking <code>boot.img</code>: the relevant changes can be found in <a href="https://github.com/KireinaHoro/preinit_angler/commit/292d47323a74c3faa2e17f6f2dfa459278af1559#diff-b67911656ef5d18c4ae36cb6741b7965">this commit</a>. The dependencies are as follows:</p>
<ul>
<li>abootimg (for parsing <code>boot.img</code> and dissecting it according to the offsets)</li>
<li>gzip, gunzip (for unzipping compressed kernel and initcpio)</li>
<li>GNU cpio version 2.12 or newer (for handling cpio archive; 2.12 is necessary as older versions don't have the <code>-D</code> option)</li>
</ul>
<h3>Pitfall on SELinux</h3>
<p>Unfortunately, using the above rules to unpack and then repack the cpio archive would result in the new <code>boot.img</code> not bootable. SELinux is denying <code>init</code> to restore SELinux contexts, causing it to exit for "Security error". Seems like Android's <code>cpio</code> implementation <code>mkbootfs</code> is somehow different from GNU cpio, and can store SELinux contexts in the cpio archive, but I failed to prove this (the <a href="https://android.googlesource.com/platform/system/core/+/master/cpio/mkbootfs.c">sources</a> for <code>mkbootfs</code> looks innocent). The fix would be switching <code>androidboot.selinux=enforcing</code> to <code>permissive</code> in the kernel command line, so that <code>init</code> can finish restoring the contexts and finish boot.</p>
<h2>Ubuntu Touch container architecture</h2>
<p>When browsing through articles related to Android and Linux working together, I ran into this article: <a href="https://wiki.ubuntu.com/Touch/ContainerArchitecture">Touch/ContainerArchitecture - Ubuntu Wiki</a>. That was actually what I'm planning about the relationship between Gentoo and Android--Android inside a container in Gentoo, except that in Ubuntu Touch, Android serves as the HAL, while in my project Android can still do whatever it want freely (of course inside its own space). A detailed explanation can be found in the "Filesystem Structure" section in <a href="https://jsteward.moe/weekly-report-0514.html">last week's report</a>.</p>
<h2>Plans for next week</h2>
<p>The next week should be about bringing Gentoo up and launching Android inside it. I've been doing some experiments this weekend about UART and launching OpenRC's <code>init</code>, and I'm running into problems. The results should fit into next week's report.</p>Gentoo GSoC Weekly Report 05/142018-05-12T00:00:00+08:002018-05-12T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-12:/weekly-report-0514.html<h2>Things done this week</h2>
<p>This week was about creating a repository for an <code>init</code> program written in C/C++, testing it out, and trying to load the real init inside Gentoo root. The progress of each subproject are listed below.</p>
<h3>Writing a "preinit" in C/C++</h3>
<p>I've set up <a href="https://github.com/KireinaHoro/preinit_angler">a …</a></p><h2>Things done this week</h2>
<p>This week was about creating a repository for an <code>init</code> program written in C/C++, testing it out, and trying to load the real init inside Gentoo root. The progress of each subproject are listed below.</p>
<h3>Writing a "preinit" in C/C++</h3>
<p>I've set up <a href="https://github.com/KireinaHoro/preinit_angler">a repository</a> called <code>preinit_angler</code>, which currently implements the test function (print haiku into kernel message so that we can verify them on next boot) described in <a href="https://jsteward.moe/replace-android-init-with-test-script.html">this article</a>. Implementing the toy init is necessary is because we set up a tiny-platform that we can place other things onto it in this way: maybe password prompts for entering LUKS passphrases, or options to choose different Android snapshots. What's more, this verifies that our toolchain was really generating correctly statically-linked executables that can be directly executed by the kernel. The repository also holds the incomplete code in attempt to implement Android <code>cryptfs</code> decryption by hand.</p>
<h3>Attempt to decrypt Android userdata</h3>
<p>I've ran into problems tackling the native encryption of Android: after spending a considerable amount of time on reading the code and splitting functions out from <code>vold</code>, I ran into the dead end of ARM's Trusted Execution Environment (TEE, learn more about it <a href="https://en.wikipedia.org/wiki/Trusted_execution_environment">here on Wikipedia</a>). The detailed process of my attempt is logged in this article: <a href="https://jsteward.moe/analysis-of-android-cryptfs.html">Analysis of Android cryptfs</a>.</p>
<h3>Bypassing Android force encryption and restructuring userdata partition</h3>
<p>As we can't use an encrypted <code>userdata</code> for now, and encryption is forced on AOSP for Nexus 6P, we'll need to bypass the policy. The last section in <a href="https://jsteward.moe/analysis-of-android-cryptfs.html">the <code>cryptfs</code> analysis article</a> ("What next?") describes how to do this. In short, we modify <code>vold</code>'s <code>fstab</code> so that <code>vold</code> won't encrypt the phone on first boot.</p>
<p>The next part is about "chainloading" the real, full-blown <code>/sbin/init</code>, which belongs to OpenRC. This is done in several stages:</p>
<ul>
<li>mounting <code>userdata</code> partition, which holds Gentoo's real filesystem root</li>
<li>switching root into the mounted partition, executing OpenRC's init</li>
<li>launching Android's init in chroot (with jail if Android init checks if it's PID1; uses <a href="https://github.com/vincentbernat/jchroot">jchroot</a>)</li>
</ul>
<p>Note that for <code>jchroot</code> to work, several namespace-related config options are needed, specifically:</p>
<div class="highlight"><pre><span></span><code>CONFIG_SYSVIPC
CONFIG_IPC_NS
CONFIG_PID_NS
CONFIG_NET_NS
CONFIG_UTS_NS
</code></pre></div>
<p>Failing to enable these options will result in <code>jchroot</code> (or to say, the <code>clone(2)</code>) failing with <code>EINVAL</code>.</p>
<h2>Current boot sequence, filesystem structure and current work status</h2>
<p>After some discussion with UnderSampled, OxR463 and heroxbd on IRC, I feel it necessary to explain a few things here.</p>
<h3>Boot sequence and where we are now</h3>
<p>The boot sequence is of the following stages:</p>
<ul>
<li>bootloader loads <code>boot.img</code>, executing the kernel;</li>
<li>kernel executes <code>/init</code>, which is <code>preinit</code>, in the initramfs that came with <code>boot.img</code>;</li>
<li><code>preinit</code> mounts <code>userdata</code> and handles mountpoints correctly, then executes Gentoo's <code>openrc-init</code>;</li>
<li>OpenRC starts, launching a service that uses <code>jchroot</code> to launch Android's init.</li>
</ul>
<p>We're currently at the <code>preinit</code> stage, trying to launch OpenRC's init.</p>
<h3>Filesystem structure</h3>
<p>The final filesystem structure would be like this: Gentoo root sits in <code>userdata</code> partition. The path <code>/var/lib/android</code> is special: it holds the Android <code>init</code> as well as Android mountpoints. When we <code>jchroot</code> into <code>/var/lib/android</code>, Android init would mount the partition it needs correctly, except for <code>/data</code> in the Android root. We'll need to change the <code>fstab</code> entry for <code>userdata</code> into a bind mount of <code>/data</code> (from Android's perspective of view; the real path in <code>userdata</code> is <code>/var/lib/android/data</code>) to <em>itself</em>, thus making <code>/data</code> a mountpoint, so as to make <code>vold</code> happy. </p>
<h2>Work to be done next week</h2>
<p>By the end of next week we shall see: an Android system successfully booting inside a chroot (and namespaces) under OpenRC init, who gets loaded by <code>preinit</code>. Several expected problems/projects:</p>
<ul>
<li><code>switch_root</code> into <code>userdata/gentoo</code> temporarily to see if OpenRC loads fine. Fix it if it's not working.</li>
<li>Refactor <code>userdata</code>, deal with the mountpoints correctly, and edit Android <code>fstab</code> accordingly. Android's <code>fstab</code> parser may not support bind mounts; we may have to bind-mount it ourselves and remove that line in <code>fstab</code> completely.</li>
<li>We'll definitely run into problems, thus a serial console through UART (in headphone jack for Nexus'es and Pixels) is necessary; asynchronous <code>console-ramoops</code> does not fulfill the requirement for interactions.</li>
</ul>
<p>This requires a headphone-jack UART cable, which was made by @imi415 on Telegram. Kudos for @imi415!</p>Analysis of Android cryptfs2018-05-11T13:00:00+08:002018-05-04T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-11:/analysis-of-android-cryptfs.html<h2>The story</h2>
<p>I'm working on my GSoC 2018 project, and part of my work at this point is to take over <code>init</code> on an Android system. In order to get to the real root located in <code>/data</code> on Android, I need to mount the encrypted Android userdata partition, which uses …</p><h2>The story</h2>
<p>I'm working on my GSoC 2018 project, and part of my work at this point is to take over <code>init</code> on an Android system. In order to get to the real root located in <code>/data</code> on Android, I need to mount the encrypted Android userdata partition, which uses Linux's <code>dm-crypt</code> target of <code>device-mapper</code>. Though I thought that it would be a pretty easy task, it turned out otherwise. In this article I analyze the routine of AOSP's <code>vold</code> decrypting the <code>userdata</code> block device.</p>
<p>Along the way of analyzing, I picked the necessary functions and macro definitions and trimmed unnecessary code in hope of splitting the <code>cryptfs</code> component out of Android's <code>vold</code>, which is too much an overkill for simply decrypting the <code>userdata</code> partition; what's more, it's virtually impossible to get <code>vold</code> working in the rural context of <code>init</code>, as nothing on the system has been set up yet. Unfortuately, a key component turned out to be rather hard to get, of which we'll see later on in this article. The partial work done can be found in <a href="https://github.com/KireinaHoro/preinit_angler">this repository</a>.</p>
<h2>Find the key function</h2>
<p>According to <a href="https://source.android.com/security/encryption/full-disk">Android Source</a>, the entire encryption/decryption logic of Android is in <code>cryptfs.cpp</code>, a component of Android's volume manager daemon <code>vold</code>. After some rough <code>C-s</code>'ing in <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp">the source</a>, we come across the key function:</p>
<div class="highlight"><pre><span></span><code><span class="k">static</span><span class="w"> </span><span class="nb nb-Type">int</span><span class="w"> </span><span class="n">test_mount_encrypted_fs</span><span class="p">(</span><span class="n">struct</span><span class="w"> </span><span class="n">crypt_mnt_ftr</span><span class="o">*</span><span class="w"> </span><span class="n">crypt_ftr</span><span class="p">,</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">passwd</span><span class="p">,</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">mount_point</span><span class="p">,</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">label</span><span class="p">)</span>
</code></pre></div>
<p>But before we look at the function body, let's check the parameters first. The definition of <code>struct crypt_mnt_ftr</code> can be found in <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.h#98"><code>cryptfs.h</code></a>. Telling from the name of the structure as well as some common sense of what should be needed to <code>test_mount</code> an encrypted filesystem, we can see that this structure is a <strong>crypto footer</strong>. As the comments in <code>cryptfs.h</code> states:</p>
<div class="highlight"><pre><span></span><code><span class="o">/*</span><span class="w"> </span><span class="n">This</span><span class="w"> </span><span class="n">structure</span><span class="w"> </span><span class="n">starts</span><span class="w"> </span><span class="mi">16</span><span class="p">,</span><span class="mi">384</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="n">before</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">end</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">hardware</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">partition</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">encrypted</span><span class="p">,</span><span class="w"> </span><span class="nb">or</span><span class="w"> </span><span class="n">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">separate</span><span class="w"> </span><span class="n">partition</span><span class="p">.</span><span class="w"> </span><span class="n">It</span><span class="o">'</span><span class="n">s</span><span class="w"> </span><span class="n">location</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">specified</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">property</span><span class="w"> </span><span class="k">set</span><span class="w"> </span><span class="n">in</span><span class="w"> </span><span class="n">init</span><span class="p">.</span><span class="o"><</span><span class="n">device</span><span class="o">></span><span class="p">.</span><span class="n">rc</span><span class="p">.</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">structure</span><span class="w"> </span><span class="n">allocates</span><span class="w"> </span><span class="mi">48</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="n">but</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nb">real</span><span class="w"> </span><span class="n">key</span><span class="w"> </span><span class="nb">size</span><span class="w"> </span><span class="n">is</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">specified</span><span class="w"> </span><span class="n">in</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nb">struct</span><span class="p">.</span><span class="w"> </span><span class="n">Currently</span><span class="p">,</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">code</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">hardcoded</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="mi">128</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">bit</span><span class="w"> </span><span class="n">keys</span><span class="p">.</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">fields</span><span class="w"> </span><span class="n">after</span><span class="w"> </span><span class="n">salt</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">only</span><span class="w"> </span><span class="n">valid</span><span class="w"> </span><span class="n">in</span><span class="w"> </span><span class="n">rev</span><span class="w"> </span><span class="mf">1.1</span><span class="w"> </span><span class="nb">and</span><span class="w"> </span><span class="n">later</span><span class="w"> </span><span class="n">stuctures</span><span class="p">.</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">Obviously</span><span class="p">,</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">does</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">include</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">last</span><span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="n">kbytes</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">partition</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">crypt_mnt_ftr</span><span class="w"> </span><span class="n">lives</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">end</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">the</span>
<span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">partition</span><span class="p">.</span>
<span class="w"> </span><span class="o">*/</span>
</code></pre></div>
<p>On Nexus 6P, the footer is located in the 16MB-<code>metadata</code> partition, while on Oneplus 5 this footer is located at the end of the <code>userdata</code> filesystem. To really know about where this information is, <code>vold</code> will read the device's <code>fstab</code> (usually located in the <code>initramfs</code>) with the help of <code>fs_mgr</code> to get the true location of the footer, be it an individual partition or an offset relative to the start of <code>userdata</code> partition.</p>
<h2>Digging deeper</h2>
<p>Now that we know where is the information that's needed to restore the structure of the userdata partition, we can start reading the function body of <code>test_mount_encrypted_fs</code>. The first (and the only one that's important) foreign function we run into is <code>decrypt_master_key</code> <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp#1610">here</a>, whose definition can be found <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp#1253">here</a>:</p>
<div class="highlight"><pre><span></span><code><span class="k">static</span><span class="w"> </span><span class="nb nb-Type">int</span><span class="w"> </span><span class="n">decrypt_master_key</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">passwd</span><span class="p">,</span><span class="w"> </span><span class="n">unsigned</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">decrypted_master_key</span><span class="p">,</span>
<span class="w"> </span><span class="n">struct</span><span class="w"> </span><span class="n">crypt_mnt_ftr</span><span class="w"> </span><span class="o">*</span><span class="n">crypt_ftr</span><span class="p">,</span>
<span class="w"> </span><span class="n">unsigned</span><span class="w"> </span><span class="nb">char</span><span class="o">**</span><span class="w"> </span><span class="n">intermediate_key</span><span class="p">,</span>
<span class="w"> </span><span class="n">size_t</span><span class="o">*</span><span class="w"> </span><span class="n">intermediate_key_size</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">kdf_func</span><span class="w"> </span><span class="n">kdf</span><span class="p">;</span>
<span class="w"> </span><span class="nb nb-Type">void</span><span class="w"> </span><span class="o">*</span><span class="n">kdf_params</span><span class="p">;</span>
<span class="w"> </span><span class="nb nb-Type">int</span><span class="w"> </span><span class="n">ret</span><span class="p">;</span>
<span class="w"> </span><span class="n">get_kdf_func</span><span class="p">(</span><span class="n">crypt_ftr</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">kdf</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">kdf_params</span><span class="p">);</span>
<span class="w"> </span><span class="n">ret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">decrypt_master_key_aux</span><span class="p">(</span><span class="n">passwd</span><span class="p">,</span><span class="w"> </span><span class="n">crypt_ftr</span><span class="o">-></span><span class="n">salt</span><span class="p">,</span><span class="w"> </span><span class="n">crypt_ftr</span><span class="o">-></span><span class="n">master_key</span><span class="p">,</span>
<span class="w"> </span><span class="n">decrypted_master_key</span><span class="p">,</span><span class="w"> </span><span class="n">kdf</span><span class="p">,</span><span class="w"> </span><span class="n">kdf_params</span><span class="p">,</span>
<span class="w"> </span><span class="n">intermediate_key</span><span class="p">,</span><span class="w"> </span><span class="n">intermediate_key_size</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">ret</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">SLOGW</span><span class="p">(</span><span class="s2">"failure decrypting master key"</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">ret</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Woah, more new friends! What comes first is <code>kdf_func</code>. After <code>ripgrep</code>'ing around, we can find that it's a <code>typedef</code> declared <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.h#230">here</a>:</p>
<div class="highlight"><pre><span></span><code><span class="n">typedef</span><span class="w"> </span><span class="nb nb-Type">int</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="n">kdf_func</span><span class="p">)(</span><span class="k">const</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">passwd</span><span class="p">,</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">unsigned</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">salt</span><span class="p">,</span>
<span class="w"> </span><span class="n">unsigned</span><span class="w"> </span><span class="nb">char</span><span class="w"> </span><span class="o">*</span><span class="n">ikey</span><span class="p">,</span><span class="w"> </span><span class="nb nb-Type">void</span><span class="w"> </span><span class="o">*</span><span class="n">params</span><span class="p">);</span>
</code></pre></div>
<p>And that <code>get_kdf_func</code> turned out to be selecting the correct Key Derivation Function that <em>derives</em> the master key that's used for encrypting the entire <code>userdata</code> function. We'll come back to the KDF later. We can see that the real cryptographical logic is in the function <code>decrypt_master_key_aux</code> <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp#1188">here</a>. The code is a little bit long to include here, and it's simply a reverse of the Key Derivation process, so instead of dissecting this function here, I'll put it off until we finish the part on KDF, after which the master key decryption process would be quite straightforward.</p>
<h2>The nightmare--Key Derivation Function</h2>
<p>Android system currently have three versions of the KDF, among which current Android version is using the <code>scrypt-keymaster</code> function defined <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp#1062">here</a>. The key derivation process is as follows:</p>
<ul>
<li>Generate 16 bytes randomly as the Disk Encryption Key (DEK) and then generate 16 bytes randomly as the salt (SALT);</li>
<li>Use scrypt (<code>crypto_scrypt</code>) on the User Password-SALT pair, resulting in a hash of 32 bytes; take this as Intermediate Key 1 (IK1)</li>
<li>Pad IK1 to match the size of secret key in crypto hardware (256 bytes RSA as of now), patching scheme as follows:
<code>00 || IK1 || 00..00 # one zero byte, 32 IK1 bytes, 223 zero bytes</code></li>
<li>Sign IK1 with crypto hardware, resulting in 256 bytes of signature as IK2;</li>
<li>Use scrypt on the IK2-SALT pair, resulting in a hash of 32 bytes; take this as IK3;</li>
<li>Use the first 16 bytes of IK3 as Key Encryption Key (KEK, used to encrypt DEK), and the last 16 bytes as Initialization Vector (IV);</li>
<li>Use AES_CBC with KEK as secret key and IV as the initialization vector to encrypt DEK, which is the encrypted master key. Store this into the data structure <code>crypt_mnt_ftr</code>, which we discussed earlier.</li>
</ul>
<p>As you may notice, the hardest part in this system to implement by hand is the <strong>crypto hardware</strong>, and the key function we can't handle here is <code>keymaster_sign_object</code>. Android HAL implements access to the crypto hardware by proxying actual operations to the vendor firmware blob. The high-level abstraction for the device is implemented <a href="https://android.googlesource.com/platform/system/vold/+/master/cryptfs.cpp#1062">here</a> as a part of <code>vold</code>, while the low-level interfaces are buried in <code>libhardware</code> component of Android, which is the Android HAL. This is the point where I stopped further investigations, as further dissecting Android HAL will take reasonably longer time, with no guarantee of successfully implementing the crypto hardware signing procedure.</p>
<h2>What next?</h2>
<p>So, decrypting Android <code>userdata</code> partition seems to be a no-go for now. Yet, fortunately, we can bypass the forced encryption by modifying Android's <code>fstab</code>. For Nexus 6P, the line for <code>userdata</code> reads:</p>
<div class="highlight"><pre><span></span><code>/dev/block/platform/soc.0/f9824900.sdhci/by-name/userdata /data ext4 noatime,nosuid,nodev,barrier=1,data=ordered,nomblk_io_submit,noauto_da_alloc,errors=panic,inode_readahead_blks=8 wait,check,forcefdeorfbe=/dev/block/platform/soc.0/f9824900.sdhci/by-name/metadata
</code></pre></div>
<p>Note the <code>forcefdeorfbe</code> keyword: this means that either Full-Disk Encryption (which is what we've discussed in this article) or File-Based Encryption is required on this device; if no encryption is present, the device will be encrypted by <code>vold</code> on boot. What's written after it is the location of the <code>crypt_mnt_ftr</code> structure we discussed above. </p>
<p>To bypass the forced encryption, we simply substitute <code>forcedfdeorfbe</code> with <code>encryptable</code> and formatting the <code>userdata</code> partition. In this way <code>userdata</code> will remain unencrypted. I'll leave encryption as something to encypt later, maybe with LUKS, which is something far easier to use in the GNU/Linux land.</p>Replace Android init with test script2018-05-05T13:00:00+08:002018-05-05T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-05:/replace-android-init-with-test-script.html<h2>Preface</h2>
<p>As a part of my GSoC 2018 project, I'll replace Android's init with one of the init systems from GNU/Linux (OpenRC to be exact), and I've been experimenting with replacing the holy PID1 on an Android system pretty long ago (dating back to 2015), though things didn't work …</p><h2>Preface</h2>
<p>As a part of my GSoC 2018 project, I'll replace Android's init with one of the init systems from GNU/Linux (OpenRC to be exact), and I've been experimenting with replacing the holy PID1 on an Android system pretty long ago (dating back to 2015), though things didn't work out at that time. According to <a href="http://whiteboard.ping.se/Android/Debian">this article</a>, replacing init is quite straightforward: one just throw in busybox along with a script named <code>init</code>, placed at the root of the initramfs, along with a suitable <code>busybox</code> binary. This article logs how I achieved this goal.</p>
<h2>Placing a hook inside Android Build System</h2>
<p>I've added a hook to manipulate the Android root (the filesystem mounted at /) before it gets packed by <code>mkbootfs</code>, a tool for packing <code>ramdisk.img</code> in the build process. The hook can be found <a href="https://github.com/KireinaHoro/android_build/commit/03ec95b81d1678d2d81b30d796a129e805ff4203">here</a>, and I'm going to edit the script that the hook calls in order to really edit the initramfs. The new contents of the script look like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">echo</span><span class="w"> </span><span class="o">=====</span><span class="w"> </span>Ramdisk<span class="w"> </span>patch<span class="w"> </span>started<span class="w"> </span>on<span class="w"> </span><span class="k">$(</span>date<span class="k">)</span><span class="w"> </span><span class="o">=====</span><span class="w"> </span>>><span class="w"> </span>~/patch_ramdisk.log
cp<span class="w"> </span>-Rv<span class="w"> </span>~/new_ramdisk/*<span class="w"> </span><span class="nv">$@</span><span class="w"> </span>>><span class="w"> </span>~/patch_ramdisk.log
<span class="nb">echo</span><span class="w"> </span><span class="o">=====</span><span class="w"> </span>Ramdisk<span class="w"> </span>successfully<span class="w"> </span><span class="nv">patched</span><span class="w"> </span><span class="o">=====</span><span class="w"> </span>>><span class="w"> </span>~/patch_ramdisk.log
</code></pre></div></td></tr></table></div>
<p>Note that this script shouldn't have any output to <code>stdout</code>, as its output gets parsed by the build system as commands; writing log directly to <code>stdout</code> would result in a build failure. We should now place the new contents of our initramfs at <code>~/new_ramdisk</code>, and its contents will get added to the final ramdisk, overriding whatever is in place.</p>
<h2>How to tell if the new init successfully gets executed</h2>
<p>This was a hard one. As we're not chainloading the Android init at this phase (it's a little bit complicated to figure out the mounts), we'll get a kernel panic right after whatever's done in our <del>fake </del>init. I thought about drawing something to the screen or activating the taptic engine in the beginning, so that we really know if the init gets executed. Yet, after in-depth discussions with @imbushuo and @Icenowy on Telegram, I realized that this was beyond my capabilities: Qualcomm's framebuffer devices require special operations to access them instead of directly reading from and writing to the device node at <code>/dev/graphics/fb0</code> (the internal display). I've tried to read the relevant parts of code from <a href="https://github.com/Tasssadar/multirom">MultiROM</a>, yet the code was complicated and doesn't compile well in my environment. The taptic engine should be something attached via GPIO on the PMIC, and we should be able to access it by writing to SPMI addresses, and, according to @imbushuo, this should be an easy task. Unfortunately, I failed to find anything useful after digging around in Nexus 6P (MSM8994)'s kernel source, and no device nodes in <code>/dev</code> look like the correct node either.</p>
<p>The <code>ramoops</code> debug facility in Android kernels came to the rescue. According to <a href="https://android.googlesource.com/kernel/common/+/android-3.18/Documentation/ramoops.txt">this in AOSP source</a>:</p>
<blockquote>
<p>Ramoops is an oops/panic logger that writes its logs to RAM before the system crashes. It works by logging oopses and panics in a circular buffer.</p>
</blockquote>
<p>The path <code>/sys/fs/pstore/console-ramoops</code> stores the kernel message buffer (also known as <code>dmesg</code>) from last kernel boot, regardless of whether the boot has succeeded, crashed (kernel panic), or the power source was cut when the system was still running (in which case the kernel may not have time to write logs back to storage). In addition, we can write to <code>dmesg</code> by writing to <code>/dev/kmsg</code>; and we can get a working <code>/dev/kmsg</code> by mounting a <code>devtmpfs</code>, when nothing has been populated yet in the new root. In this way, our new init can be made like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span><code><span class="gh">#</span>!/sbin/bb
<span class="gh">#</span> write log to kmsg
log() {
echo new_era: $@ > /dev/kmsg
}
<span class="gh">#</span> set up psuedo-filesystems
mount -t devtmpfs devtmpfs /dev
<span class="gh">#</span> write the haiku
log Old pond
log Frog jumps in
log Sound of water
log -- Matsuo Basho
<span class="gh">#</span> sleep for a while
sleep 10
</code></pre></div></td></tr></table></div>
<p>Get a statically-linked busybox for aarch64, and put it under <code>sbin/</code> in the new ramdisk. Structure the initramfs so that it looks like the following:</p>
<div class="highlight"><pre><span></span><code><span class="n">jsteward</span><span class="nv">@yuki</span><span class="err">:</span><span class="o">~/</span><span class="n">new_ramdisk</span><span class="err">$</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="o">*</span>
<span class="o">-</span><span class="n">rwxr</span><span class="o">-</span><span class="n">xr</span><span class="o">-</span><span class="n">x</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="mi">261</span><span class="w"> </span><span class="n">May</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">02</span><span class="err">:</span><span class="mi">48</span><span class="w"> </span><span class="n">init</span>
<span class="nl">dev</span><span class="p">:</span>
<span class="n">total</span><span class="w"> </span><span class="mi">0</span>
<span class="k">proc</span><span class="err">:</span>
<span class="n">total</span><span class="w"> </span><span class="mi">0</span>
<span class="nl">sbin</span><span class="p">:</span>
<span class="n">total</span><span class="w"> </span><span class="mi">1764</span>
<span class="n">lrwxrwxrwx</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">May</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">02</span><span class="err">:</span><span class="mi">49</span><span class="w"> </span><span class="n">bb</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">busybox</span>
<span class="o">-</span><span class="n">rwxr</span><span class="o">-</span><span class="n">xr</span><span class="o">-</span><span class="n">x</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="mi">1805928</span><span class="w"> </span><span class="n">May</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">00</span><span class="err">:</span><span class="mi">42</span><span class="w"> </span><span class="n">busybox</span>
<span class="n">lrwxrwxrwx</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">May</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">02</span><span class="err">:</span><span class="mi">49</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">busybox</span>
<span class="n">lrwxrwxrwx</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">May</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">02</span><span class="err">:</span><span class="mi">49</span><span class="w"> </span><span class="n">reboot</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">busybox</span>
<span class="n">lrwxrwxrwx</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="n">jsteward</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">May</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">02</span><span class="err">:</span><span class="mi">49</span><span class="w"> </span><span class="n">sleep</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">busybox</span>
<span class="n">jsteward</span><span class="nv">@yuki</span><span class="err">:</span><span class="o">~/</span><span class="n">new_ramdisk</span><span class="err">$</span>
</code></pre></div>
<p>And, build AOSP again, and take <code>out/target/product/angler/boot.img</code>. Reboot the phone into fastboot mode, and issue the following to let it boot the new <code>boot.img</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">jsteward</span><span class="nv">@yuki</span><span class="err">:</span><span class="o">~</span><span class="err">$</span><span class="w"> </span><span class="n">fastboot</span><span class="w"> </span><span class="n">boot</span><span class="w"> </span><span class="n">boot</span><span class="p">.</span><span class="n">img</span>
<span class="p">(</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="k">output</span><span class="w"> </span><span class="n">elided</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="p">)</span>
</code></pre></div>
<p>The phone should load the new kernel and initramfs (it's slow--be patient) and reboot in approximately 30 seconds. When it boots back into Android, we can then check <code>/sys/fs/pstore/console-ramoops</code> to see if our haiku got in there:</p>
<div class="highlight"><pre><span></span><code><span class="n">jsteward</span><span class="nv">@yuki</span><span class="err">:</span><span class="o">~</span><span class="err">$</span><span class="w"> </span><span class="n">adb</span><span class="w"> </span><span class="n">root</span>
<span class="n">restarting</span><span class="w"> </span><span class="n">adbd</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">root</span>
<span class="n">jsteward</span><span class="nv">@yuki</span><span class="err">:</span><span class="o">~</span><span class="err">$</span><span class="w"> </span><span class="n">adb</span><span class="w"> </span><span class="n">shell</span>
<span class="nl">angler</span><span class="p">:</span><span class="o">/</span><span class="err">#</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="o">-</span><span class="n">A2</span><span class="w"> </span><span class="n">new_era</span><span class="w"> </span><span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">fs</span><span class="o">/</span><span class="n">pstore</span><span class="o">/</span><span class="n">console</span><span class="o">-</span><span class="n">ramoops</span>
<span class="o">[</span><span class="n"> 12.474410</span><span class="o">]</span><span class="w"> </span><span class="nl">new_era</span><span class="p">:</span><span class="w"> </span><span class="k">Old</span><span class="w"> </span><span class="n">pond</span>
<span class="o">[</span><span class="n"> 12.478773</span><span class="o">]</span><span class="w"> </span><span class="nl">new_era</span><span class="p">:</span><span class="w"> </span><span class="n">Frog</span><span class="w"> </span><span class="n">jumps</span><span class="w"> </span><span class="ow">in</span>
<span class="o">[</span><span class="n"> 12.492715</span><span class="o">]</span><span class="w"> </span><span class="nl">new_era</span><span class="p">:</span><span class="w"> </span><span class="n">Sound</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">water</span>
<span class="o">[</span><span class="n"> 12.496127</span><span class="o">]</span><span class="w"> </span><span class="nl">new_era</span><span class="p">:</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="n">Matsuo</span><span class="w"> </span><span class="n">Basho</span>
<span class="o">[</span><span class="n"> 22.520767</span><span class="o">]</span><span class="w"> </span><span class="n">Kernel</span><span class="w"> </span><span class="n">panic</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="nl">syncing</span><span class="p">:</span><span class="w"> </span><span class="n">Attempted</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">kill</span><span class="w"> </span><span class="n">init</span><span class="err">!</span><span class="w"> </span><span class="n">exitcode</span><span class="o">=</span><span class="mh">0x00000000</span>
<span class="o">[</span><span class="n"> 22.520767</span><span class="o">]</span>
<span class="nl">angler</span><span class="p">:</span><span class="o">/</span><span class="err">#</span>
</code></pre></div>
<p>Note the 10 second delay in <code>dmesg</code>: that's when our init is sleeping for 10 seconds; kernel panics right away when init dies, which was the expected behavior--a real init should never die.</p>
<h2>File permissions in ramdisk</h2>
<p>I was stuck with kernel couldn't execute my init in my first few attempts. The log looked like this:</p>
<div class="highlight"><pre><span></span><code>[ 11.984835] Failed to execute /init
[ 11.988149] Kernel panic - not syncing: No init (further output elided...)
</code></pre></div>
<p>And, thanks to the help from @Icenowy, I discovered that the problem was I didn't put <code>busybox</code> in <code>/sbin</code> in the first time, and the build system didn't give it the executable bit in the ramdisk (despite that it had the executable bit in the working folder). As <a href="https://android.googlesource.com/platform/system/core/+/master/libcutils/fs_config.cpp#194">the source</a> states, <code>init*</code> are in the permission set <code>AID_ROOT</code>, and <code>sbin/*</code> are in the permission set <code>AID_SHELL</code>. Both permission sets have the executable bit, so placing <code>busybox</code> in <code>sbin/</code> should solve the problem, and it did solve the problem.</p>Installing Gentoo in Hyper-V2018-05-05T00:00:00+08:002018-05-05T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-05:/install-gentoo-in-hyper-v.html<h2>The story</h2>
<p>After struggling with HiDPI issues on native GNU/Linux and battery life issues on macOS, I resorted to using Windows 10 as the main operating system on my laptop. Though there's WSL, it's not running Gentoo and has a severely degraded performance. As <code>crossdev</code> is really easy to …</p><h2>The story</h2>
<p>After struggling with HiDPI issues on native GNU/Linux and battery life issues on macOS, I resorted to using Windows 10 as the main operating system on my laptop. Though there's WSL, it's not running Gentoo and has a severely degraded performance. As <code>crossdev</code> is really easy to use for building a cross-compile toolchain (<a href="https://jsteward.moe/weekly-report-0507.html">the reason that I needed a cross-compile toolchain</a>) with <a href="https://wiki.gentoo.org/wiki/Crossdev">crossdev</a>. So, I need a virtual machine running Gentoo on my Windows 10 system. This article logs the process of installing Gentoo inside Hyper-V, Windows 10's built-in hypervisor.</p>
<h2>Enable Hyper-V on Windows 10</h2>
<p>First of all, we need to enable Hyper-V on our Windows 10 host. With Hyper-V, the underlying structure of Windows changes greatly; read <a href="https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/hyper-v-architecture">this</a> for a detailed explanation of the Hyper-V architecture. Navigate to Control Panel -> Programs and Features -> Enable or disable Windows features (my system is in Japanese, so this maybe inaccurate; follow what's shown on your system), where you'll see an item called Hyper-V. Tick both "Hyper-V platform" and "Hyper-V management tools". Note that you'll need at least the Pro x64 version of Windows to see that "Hyper-V platform" feature. Reboot after committing the changes.</p>
<h2>Downloading the install ISO and creating the virtual machine</h2>
<p>We need the minimal ISO for amd64 to install. Navigate to your favorite mirror site and pick up the ISO image in <code>releases/amd64/autobuilds/current-install-amd64-minimal/</code>. The link I used is <a href="https://mirrors.tuna.tsinghua.edu.cn/gentoo/releases/amd64/autobuilds/current-install-amd64-minimal/install-amd64-minimal-20180415T214502Z.iso">here</a> (hosted by TUNA Association of Tsinghua University).</p>
<p>Locate and open the Hyper-V Manager on Windows (hint: use Windows Search). Create a virtual machine from Operations (right sidebar) -> New -> Virtual Machine. (Again, this may not be what's exactly displayed on the screen; follow what you see that has the most similar meanings.) Follow the wizard to create the virtual machine. When you're prompted to select the Generation of the new VM, choose Gen 1, which means BIOS boot--UEFI is an overkill here; when prompted for CD/DVD drive, choose the ISO you just downloaded. As this is a Gentoo guest, I would recommend as much RAM as possible (I gave 16GB out of a total physical RAM of 32GB--your mileage may vary). The VM should now be successfully created.</p>
<p>Before starting the virtual machine, we need to configure several things. Right click on the newly-created virtual machine and select Settings.</p>
<ul>
<li>Select the Processor page from the left sidebar, and adjust the number of cores you wish to grant to the Gentoo guest. </li>
<li>Select the Network page from the left sidebar, and choose the adapter type. The default switch is configured to do NAT for the virtual machine. If you want to access the virtual machine from outside the host OS, create a new Switch by selecting the Virtual Switch Manager from the right sidebar in the Hyper-V Manager main window and creating a new switch that bridges the virtual machines' NICs with the interface you choose on the host.</li>
</ul>
<p><strong>NOTE:</strong> for Gen 1 virtual machines to boot, the boot drive (virtual disk on which the OS sits) has to be an IDE drive. SCSI drives won't work as boot drives; they'll function as data drives only. Double-check your configuration to make sure that you'll install Gentoo (at least the bootloader) on an IDE drive, otherwise you'll encounter cryptic errors (such as garbled text in VM firmware).</p>
<h2>Boot the virtual machine and install system</h2>
<p>Double-click on the configured virtual machine in Hyper-V Manager, then choose Start. Install Gentoo as usual (<a href="https://wiki.gentoo.org/wiki/Handbook:AMD64">AMD64 Handbook</a> for reference). If the Hyper-V console was too cumbersome to use, enable sshd in LiveCD and connect to the virtual machine via XShell, PuTTY, or any SSH client of your choice. Check the VM's ip address via <code>ip addr</code>; depending on your virtual machine network configuration, you may see a NAT address (<code>172.17.0.0/16</code>) or an address from your upstream network (if in bridged mode). Connect to the address the VM got.</p>
<p>When configuring the kernel, note that the following options are needed for Hyper-V guest support; otherwise the kernel may not be able to find the root partition or other devices.</p>
<div class="highlight"><pre><span></span><code>CONFIG_HYPERV=y
CONFIG_HYPERV_STORAGE=y
CONFIG_HYPERV_NET=y
CONFIG_HYPERV_KEYBOARD=y
CONFIG_HYPERV_UTILS=y
CONFIG_HYPERV_BALLOON=y
CONFIG_FB_HYPERV=y
CONFIG_HID_HYPERV_MOUSE=y
</code></pre></div>
<p>I prefer a minimal kernel config, to speed up build time and to reduce final kernel image size. <a href="https://gist.github.com/KireinaHoro/cfa71bb97e8e893ac854407b151ae360">This</a> is the config I use, with most unneeded drivers removed.</p>
<p>For bootloader, I've picked LILO, as it has the <code>run-parts</code> plugin for kernel install, and LILO itself is very lightweight and easy to configure. Refer to <a href="https://wiki.gentoo.org/wiki/LILO">LILO's Gentoo Wiki page</a> for information about how to install and configure.</p>
<h2>Bootstrap a aarch64 cross-compile toolchain</h2>
<p>This is the point of having Gentoo as the guest OS, instead of just using Ubuntu in WSL or some other distribution. Emerge <code>crossdev</code> and build a cross-compile toolchain for <code>CHOST=aarch64-unknown-linux-gnu</code>. Note that before using crossdev, we'll need an overlay to store the modified ebuilds in. Consult <a href="https://wiki.gentoo.org/wiki/Custom_repository#Crossdev">this article</a> for how to setup an overlay for <code>crossdev</code>.</p>
<div class="highlight"><pre><span></span><code>localhost / # emerge -v crossdev
localhost / # crossdev -S -P -v -t aarch64
</code></pre></div>
<p>Verify that the newly-created toolchain works fine by compiling a simple "Hello, world!" program and executing it on the target machine:</p>
<div class="highlight"><pre><span></span><code><span class="n">localhost</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="n">cat</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">hello</span><span class="p">.</span><span class="n">cc</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">EOF</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><iostream></span>
<span class="kr">int</span><span class="w"> </span><span class="n">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"Hello, world!"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">EOF</span>
<span class="n">localhost</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="n">aarch64</span><span class="o">-</span><span class="n">unknown</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">-</span><span class="n">g</span><span class="o">++</span><span class="w"> </span><span class="n">hello</span><span class="p">.</span><span class="n">cc</span><span class="w"> </span><span class="o">-</span><span class="n">o</span><span class="w"> </span><span class="n">hello</span><span class="w"> </span><span class="o">-</span><span class="kr">static</span>
<span class="n">localhost</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="n">hello</span>
<span class="nl">hello</span><span class="p">:</span><span class="w"> </span><span class="n">ELF</span><span class="w"> </span><span class="mi">64</span><span class="o">-</span><span class="nf">bit</span><span class="w"> </span><span class="n">LSB</span><span class="w"> </span><span class="n">executable</span><span class="p">,</span><span class="w"> </span><span class="n">ARM</span><span class="w"> </span><span class="n">aarch64</span><span class="p">,</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">(</span><span class="n">GNU</span><span class="o">/</span><span class="n">Linux</span><span class="p">),</span><span class="w"> </span><span class="n">statically</span><span class="w"> </span><span class="n">linked</span><span class="p">,</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">GNU</span><span class="o">/</span><span class="n">Linux</span><span class="w"> </span><span class="mf">3.7.0</span><span class="p">,</span><span class="w"> </span><span class="o">not</span><span class="w"> </span><span class="n">stripped</span>
<span class="n">localhost</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="err">#</span>
</code></pre></div>
<p>Transfer the <code>hello</code> executable to the target machine and see if it works.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>adb<span class="w"> </span>push<span class="w"> </span>hello<span class="w"> </span>/sdcard/
<span class="o">(</span><span class="w"> </span>...<span class="w"> </span>output<span class="w"> </span>elided<span class="w"> </span>...<span class="w"> </span><span class="o">)</span>
$<span class="w"> </span>adb<span class="w"> </span>shell
angler:/#<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>/sdcard
angler:/sdcard#<span class="w"> </span>./hello
Hello,<span class="w"> </span>world!
angler:/sdcard#
</code></pre></div>Building a Gentoo chroot in Android2018-05-04T11:00:00+08:002018-05-05T23:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-04:/building-gentoo-chroot-in-android.html<h2>Preface</h2>
<p>Chrooting is usually a key part in installing a GNU/Linux system, and there's no difference here in my GSoC 2018 work. In this article we'll build a Gentoo chroot with <code>CHOST=aarch64-unknown-linux-gnu</code>, set up ssh connection to it, and install some necessary tools.</p>
<h2>Preparation work</h2>
<p>First of all …</p><h2>Preface</h2>
<p>Chrooting is usually a key part in installing a GNU/Linux system, and there's no difference here in my GSoC 2018 work. In this article we'll build a Gentoo chroot with <code>CHOST=aarch64-unknown-linux-gnu</code>, set up ssh connection to it, and install some necessary tools.</p>
<h2>Preparation work</h2>
<p>First of all, we need to find somewhere to store our chroot. Connect the phone to the computer, authorize adb access on the phone, then issue the following:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>adb<span class="w"> </span>root<span class="w"> </span><span class="c1"># remember to enable root access for adb in Developer Options</span>
restarting<span class="w"> </span>adbd<span class="w"> </span>as<span class="w"> </span>root
$<span class="w"> </span>adb<span class="w"> </span>shell
angler:/<span class="w"> </span><span class="c1"># mkdir -p /data/gentoo && cd /data/gentoo</span>
angler:/data/gentoo<span class="w"> </span><span class="c1">#</span>
</code></pre></div>
<p>Then, we need to fetch the <code>stage3</code> image corresponding to the architecture--<code>aarch64</code> for Nexus 6P. Currently, <code>aarch64</code> stages are not stable yet. The current status of Gentoo's ARM64 Project can be found <a href="https://wiki.gentoo.org/wiki/Project:ARM64">here</a>. According to the developers on <a href="mailto:gentoo-dev@lists.gentoo.org">gentoo-dev</a> in <a href="https://archives.gentoo.org/gentoo-dev/message/9f25d996bcc00e318bdfa2ec4e071be4">this email</a>, despite the fact that arm64 is still experimental for Gentoo, development is definitely taking place and we're getting packages keyworded every day. Consequently, arm64 seems to be a better choice than arm.</p>
<p>As no arm64 profiles are stable as of writing, I'll use the experimental stages, and they're located in <code>experimental/arm64</code> in the Gentoo mirror you choose. The link I used is <a href="https://mirrors.tuna.tsinghua.edu.cn/gentoo/experimental/arm64/stage3-arm64-20180305.tar.bz2">here</a>. You can either download the tarball on your device directly, or transfer it to the phone via <code>adb push</code> after downloading it on your computer. Move the tarball to <code>/data/gentoo/</code> for the next step.</p>
<h2>Unpack tarball, mount filesystems, and chroot</h2>
<p>Unpack the tarball:</p>
<div class="highlight"><pre><span></span><code><span class="n">angler</span><span class="p">:</span><span class="o">/</span><span class="w"> </span><span class="c1"># cd /data/gentoo</span>
<span class="n">angler</span><span class="p">:</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">gentoo</span><span class="w"> </span><span class="c1"># tar xvf stage3-*.tar.bz2</span>
<span class="p">(</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="n">lines</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="n">elided</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">)</span>
<span class="n">angler</span><span class="p">:</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">gentoo</span><span class="w"> </span><span class="c1"># ls</span>
<span class="n">bin</span><span class="w"> </span><span class="n">boot</span><span class="w"> </span><span class="n">dev</span><span class="w"> </span><span class="n">etc</span><span class="w"> </span><span class="n">home</span><span class="w"> </span><span class="n">lib</span><span class="w"> </span><span class="n">lib64</span><span class="w"> </span><span class="n">media</span><span class="w"> </span><span class="n">mnt</span><span class="w"> </span><span class="n">opt</span><span class="w"> </span><span class="n">proc</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="n">sbin</span><span class="w"> </span><span class="n">sys</span><span class="w"> </span><span class="n">tmp</span><span class="w"> </span><span class="n">usr</span><span class="w"> </span><span class="k">var</span>
<span class="n">angler</span><span class="p">:</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">gentoo</span><span class="w"> </span><span class="c1">#</span>
</code></pre></div>
<p>Clean up the bogus device nodes in stage3 tarball, and mount the necessary psuedo-filesystems:</p>
<div class="highlight"><pre><span></span><code><span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">rm</span><span class="w"> </span><span class="nt">-rf</span><span class="w"> </span><span class="nt">dev</span><span class="o">/*</span><span class="w"> </span><span class="nt">tmp</span><span class="o">/*</span>
<span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">a</span><span class="w"> </span><span class="nt">in</span><span class="w"> </span><span class="nt">proc</span><span class="w"> </span><span class="nt">sys</span><span class="w"> </span><span class="nt">dev</span><span class="o">;</span><span class="w"> </span><span class="nt">do</span><span class="w"> </span><span class="nt">mount</span><span class="w"> </span><span class="nt">--rbind</span><span class="w"> </span><span class="p">{</span><span class="err">/,</span><span class="p">}</span><span class="o">$</span><span class="nt">a</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nt">mount</span><span class="w"> </span><span class="nt">--make-rslave</span><span class="w"> </span><span class="o">$</span><span class="nt">a</span><span class="o">;</span><span class="w"> </span><span class="nt">done</span>
<span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">mkdir</span><span class="w"> </span><span class="nt">-p</span><span class="w"> </span><span class="nt">dev</span><span class="o">/</span><span class="nt">shm</span>
<span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">chmod</span><span class="w"> </span><span class="nt">777</span><span class="w"> </span><span class="nt">dev</span><span class="o">/</span><span class="nt">shm</span>
<span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">mount</span><span class="w"> </span><span class="nt">-t</span><span class="w"> </span><span class="nt">tmpfs</span><span class="w"> </span><span class="nt">tmpfs</span><span class="w"> </span><span class="nt">dev</span><span class="o">/</span><span class="nt">shm</span>
<span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">mount</span><span class="w"> </span><span class="nt">-t</span><span class="w"> </span><span class="nt">devpts</span><span class="w"> </span><span class="nt">-o</span><span class="w"> </span><span class="nt">gid</span><span class="o">=</span><span class="nt">5</span><span class="w"> </span><span class="nt">devpts</span><span class="w"> </span><span class="nt">dev</span><span class="o">/</span><span class="nt">pts</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">requirement</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">portage</span><span class="err">'</span><span class="nt">s</span><span class="w"> </span><span class="nt">configure</span>
<span class="nt">angler</span><span class="o">:/</span><span class="nt">data</span><span class="o">/</span><span class="nt">gentoo</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nt">mount</span><span class="w"> </span><span class="nt">-t</span><span class="w"> </span><span class="nt">tmpfs</span><span class="w"> </span><span class="nt">tmpfs</span><span class="w"> </span><span class="nt">tmp</span>
</code></pre></div>
<p>Change root into <code>/data/gentoo</code>, set up timezone, <code>resolv.conf</code>, ssh public keys, and edit <code>make.conf</code> so we have an environment that works properly.</p>
<div class="highlight"><pre><span></span><code>angler:/data/gentoo # chroot . /bin/su
localhost / # echo "nameserver 101.6.6.6" >> /etc/resolv.conf
localhost / # echo "Asia/Shanghai" > /etc/timezone
localhost / # emerge-webrsync
( ... lines of output elided ... )
localhost / # emerge --config timezone-data
( ... lines of output elided ... )
localhost / # emerge -v mosh --autounmask-write
( ... lines of output elided ... )
localhost / # ssh-keygen -A
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
</code></pre></div>
<p>We can now connect to the chroot environment via <code>mosh</code>. For ease of starting the environment, we create the following script <code>/data/gentoo/start_chroot</code>:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/system/bin/sh</span>
mount<span class="w"> </span>--rbind<span class="w"> </span>/proc<span class="w"> </span>/data/gentoo/proc
mount<span class="w"> </span>--make-rslave<span class="w"> </span>/data/gentoo/proc
mount<span class="w"> </span>--rbind<span class="w"> </span>/sys<span class="w"> </span>/data/gentoo/sys
mount<span class="w"> </span>--make-rslave<span class="w"> </span>/data/gentoo/sys
mount<span class="w"> </span>--rbind<span class="w"> </span>/dev<span class="w"> </span>/data/gentoo/dev
mount<span class="w"> </span>--make-rslave<span class="w"> </span>/data/gentoo/dev
mkdir<span class="w"> </span>-p<span class="w"> </span>/data/gentoo/dev/shm
chmod<span class="w"> </span><span class="m">777</span><span class="w"> </span>/data/gentoo/dev/shm
mount<span class="w"> </span>-t<span class="w"> </span>tmpfs<span class="w"> </span>tmpfs<span class="w"> </span>/data/gentoo/dev/shm
mount<span class="w"> </span>-t<span class="w"> </span>devpts<span class="w"> </span>-o<span class="w"> </span><span class="nv">gid</span><span class="o">=</span><span class="m">5</span><span class="w"> </span>devpts<span class="w"> </span>/data/gentoo/dev/pts
mount<span class="w"> </span>-t<span class="w"> </span>tmpfs<span class="w"> </span>tmpfs<span class="w"> </span>/data/gentoo/tmp
<span class="nb">exec</span><span class="w"> </span>chroot<span class="w"> </span>/data/gentoo<span class="w"> </span>/usr/sbin/sshd
</code></pre></div>
<p>After the system reboots, we can issue the following to bring up the sshd inside the chroot:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>adb<span class="w"> </span>root
restarting<span class="w"> </span>adbd<span class="w"> </span>as<span class="w"> </span>root
$<span class="w"> </span>adb<span class="w"> </span>shell<span class="w"> </span>/data/gentoo/start_chroot
$
</code></pre></div>
<p>And, as a bonus, we can restart <code>adbd</code> on the phone to kill all process spawned by it, including the entire chroot:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>adb<span class="w"> </span>unroot
</code></pre></div>Gentoo GSoC Weekly Report 05/072018-05-04T11:00:00+08:002018-05-04T11:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-05-04:/weekly-report-0507.html<h2>Preface</h2>
<p>This week was about getting the Gentoo stage3 to work on Android (as a subsystem of Android currently), and replacing init located in the initramfs, which was part of the <code>boot.img</code> that was used to boot the system. To ease the process of compiling for a different architecture …</p><h2>Preface</h2>
<p>This week was about getting the Gentoo stage3 to work on Android (as a subsystem of Android currently), and replacing init located in the initramfs, which was part of the <code>boot.img</code> that was used to boot the system. To ease the process of compiling for a different architecture, I installed a Gentoo guest in Hyper-V on my laptop and built a cross-compile toolchain for <code>aarch64-unknown-linux-gnu</code> in it. As the three things does not relate to each other very much, I'm splitting them into three separate articles, and placing the link below for reference.</p>
<ul>
<li><a href="https://jsteward.moe/building-gentoo-chroot-in-android.html">Building a Gentoo chroot in Android</a></li>
<li><a href="https://jsteward.moe/install-gentoo-in-hyper-v.html">Installing Gentoo in Hyper-V</a></li>
<li><a href="https://jsteward.moe/replace-android-init-with-test-script.html">Replace Android init with test script</a></li>
</ul>Gentoo GSoC Weekly Report 04/302018-04-30T12:00:00+08:002018-04-30T16:30:00+08:00Pengcheng Xutag:jsteward.moe,2018-04-30:/weekly-report-0430.html<h2>Preface</h2>
<p>According to the schedule in the proposal, this week was about getting familiar with Lineage OS building process and installing it on the development device. This blog post summarizes the work I've done and the problems I ran into (and successfully fixed) in the process, and makes a brief …</p><h2>Preface</h2>
<p>According to the schedule in the proposal, this week was about getting familiar with Lineage OS building process and installing it on the development device. This blog post summarizes the work I've done and the problems I ran into (and successfully fixed) in the process, and makes a brief plan about what will be done in the following week.</p>
<h2>Work done this week</h2>
<h3>Setting up working environment</h3>
<p>The working environemnt I'm currently using is of two parts:</p>
<ul>
<li>laptop running Windows / macOS for flashing the ROM package onto device</li>
<li>server running Linux for building and editting sources</li>
</ul>
<p>For connection to fastboot environment, I had to install the Android USB drivers provided by Google available for download <a href="https://developer.android.com/studio/run/win-usb">here</a>.</p>
<p>The build machine I use runs Ubuntu 16.04.3, which does not require much configuration. After pulling the Lineage OS source tree, I was able to complete a build according to the build instructions given by <a href="https://wiki.lineageos.org/devices/angler/build">Lineage OS wiki</a>.</p>
<h3>Modifying kernel tree in Lineage OS build system</h3>
<p>The workflow introduced in this section is for integrating the process of building kernel into the process of building the entire Android system. Compiling the kernel separately (i.e. configuring the cross-compile toolchain manually) will not require this specific workflow.</p>
<p>The kernel tree for the device I'm using for development (Nexus 6P, codename <code>angler</code>) has its kernel sources stored at </p>
<div class="highlight"><pre><span></span><code>kernel/huawei/angler
</code></pre></div>
<p>A correct workflow for the changes to the kernel source to be considered when building the entire Android system via <code>brunch angler</code> requires one to commit changes he made; otherwise the build process will fail, complaining that the work tree is not clean. To clean the kernel repository, execute the following command:</p>
<div class="highlight"><pre><span></span><code>croot && cd kernel/huawei/angler && git clean -xfd
</code></pre></div>
<p>To modify the kernel configuration properly, one need to start with one of the <code>defconfig</code>s that define devices, located in</p>
<div class="highlight"><pre><span></span><code><span class="cp">${</span><span class="n">KERNEL_SOURCE</span><span class="cp">}</span>/arch/<span class="cp">${</span><span class="n">ARCH</span><span class="cp">}</span>/configs/<span class="cp">${</span><span class="n">BOARD_NAME</span><span class="cp">}</span>_defconfig
</code></pre></div>
<p>In this specific case, the configuration I use is</p>
<div class="highlight"><pre><span></span><code><span class="cp">${</span><span class="n">KERNEL_SOURCE</span><span class="cp">}</span>/arch/arm64/configs/lineageos_angler_defconfig
</code></pre></div>
<p>One can issue <code>make ${BOARD_NAME}_defconfig</code> followed by <code>make menuconfig</code> to customize the kernel configurations. To build the kernel with the configuration tailored to one's needs, he copies the generated <code>.config</code> file to override the <code>defconfig</code> he picks. Though the generated <code>.config</code> file will be far longer than the <code>defconfig</code> file, which exactly defines what's needed for the board's functionality, the configuration generated this way will work as well. The Android build system then picks the configuration for the device to build the kernel. Note that cleaning the repository is required, as stated above.</p>
<p>Some compile errors (such as missing semi-colons and <code>#include</code>'s) pop up as I enable options in kernel config. <del>(This clearly demonstrates Google's poor code quality :/ )</del> I've forked the kernel repository <a href="https://github.com/KireinaHoro/android_kernel_huawei_angler/tree/tweak-config">on GitHub</a> to store my changes to the kernel source.</p>
<h3>Flashing the compiled ROM package onto the device</h3>
<p>I've been using Lineage OS before I switched over to Android Beta since the release of Oreo, so TWRP is present on the device. Yet, due to unknown reasons, the updater binary of Lineage OS ROM package had trouble identifying my device as an <code>angler</code> device correctly. It kept reporting:</p>
<div class="highlight"><pre><span></span><code><span class="nx">This</span><span class="w"> </span><span class="kn">package</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">device</span><span class="p">:</span><span class="w"> </span><span class="nx">angler</span><span class="p">;</span><span class="w"> </span><span class="nx">this</span><span class="w"> </span><span class="nx">device</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="p">.</span>
</code></pre></div>
<p>There should be a device name (e.g. <code>angler</code>) before that end mark in the above error message. Seems like the property strings that describe the device are not properly read by the update binary. In fact, the code that generated the above error message (in <code>META-INF/com/google/android/updater-script</code> in the ROM package) reads:</p>
<div class="highlight"><pre><span></span><code><span class="nx">assert</span><span class="p">(</span><span class="nx">getprop</span><span class="p">(</span><span class="s">"ro.product.device"</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">"angler"</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nx">getprop</span><span class="p">(</span><span class="s">"ro.build.product"</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">"angler"</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nx">abort</span><span class="p">(</span><span class="s">"E3004: This package is for device: angler; this device is "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">getprop</span><span class="p">(</span><span class="s">"ro.product.device"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">"."</span><span class="p">););</span>
</code></pre></div>
<p>The issue was later resolved by flashing the newest version of TWRP (version 3.2.1-0 for <code>angler</code> at the time of writing).</p>
<p>Another problem is that the old system (the Android Beta Program build) has encrypted the <code>userdata</code> partition with LUKS, resulting in the recovery and the newly-installed system consistently asking for a password to decrypt the partition. As formatting the <code>userdata</code> partition after decrypting the partition in recovery only formats the <code>ext4</code> filesystem, which was <em>on top of</em> the LUKS partition, I had to wipe the LUKS header in the partition manually in recovery as instructed <a href="https://android.stackexchange.com/questions/98228/removing-encryption-from-recovery">here</a>. The key is to execute the following in recovery shell:</p>
<div class="highlight"><pre><span></span><code><span class="nv">dd</span><span class="w"> </span><span class="k">if</span><span class="o">=/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">zero</span><span class="w"> </span><span class="nv">of</span><span class="o">=/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">block</span><span class="o">/</span><span class="nv">platform</span><span class="o">/</span><span class="nv">soc</span>.<span class="mi">0</span><span class="o">/</span><span class="nv">f9824900</span>.<span class="nv">sdhci</span><span class="o">/</span><span class="nv">by</span><span class="o">-</span><span class="nv">name</span><span class="o">/</span><span class="nv">userdata</span><span class="w"> </span><span class="nv">bs</span><span class="o">=</span><span class="mi">4096</span><span class="w"> </span><span class="nv">count</span><span class="o">=</span><span class="mi">512</span>
</code></pre></div>
<p>Make sure that the partition (block device specified after <code>of=</code>) is the correct one. Accidentally wiping critical partitions (e.g. partitions that store the bootloader) can result in a <strong>HARD BRICK</strong> which would require Qualcomm's tools to repair, and that will be an extremely complicated process.</p>
<h3>Explorations in the Android Build System</h3>
<p>After <del>ripgrep'ing</del>examining the AOSP source tree, I've discovered the following information that will be useful for further progress:</p>
<ul>
<li>source code for Android init in <code>system/core/init/</code>; written in C++</li>
<li>packed initramfs image <code>ramdisk.img</code> (gzip-compressed cpio archive, without kernel) at <code>out/target/product/angler/ramdisk.img</code></li>
<li>source directory for packing the initramfs in <code>boot.img</code> at <code>out/target/product/angler/root/</code></li>
<li>Makefile for generating the <code>ninja</code> rules responsible for packing initramfs at <code>build/make/core/Makefile</code></li>
</ul>
<p>I've added a hook to the Makefile above, which runs before the image gets packed (by the tool <code>mkbootimg</code>), and after the contents of the initramfs get populated. The hook is a bash script at <code>build/make/core/pre_ramdisk_hook.sh</code>. The forked <code>android_build</code> repository is <a href="https://github.com/KireinaHoro/android_build/tree/ramdisk-hook">here</a>, and the related commit is <a href="https://github.com/KireinaHoro/android_build/commit/03ec95b81d1678d2d81b30d796a129e805ff4203">03ec95b</a>.</p>
<h3>Android persistent store (previously known as last_kmg)</h3>
<p>Android had a place to store the kernel message buffer for view on the next boot, which is useful for debugging boot failures. The location for the store used to be <code>/proc/last_kmsg</code>, but it has changed to the following location since Android 6.0:</p>
<div class="highlight"><pre><span></span><code>/sys/fs/pstore/console-ramoops
</code></pre></div>
<h2>Plans for next week</h2>
<p>The next week should be about getting a stage3 onto the Nexus 6P. As the current stage3 tarballs for <code>arm</code> looks quite outdated (20161129), I expect that some efforts will be needed to roll it up to match the most up-to-date portage tree.</p>
<p>Another task is to write a small test program that makes some noise (e.g. printing something to dmesg, firing up the taptic engine, or draw something on the screen) upon being executed. By replacing Android init with this program, we can check if home-brewed executables can be run properly (i.e. SELinux is not interfering with its execution). And, if that works as expected, I can then start reading the code of Android init, to learn about how it mounts the encrypted partitions, in order to get ready for loading the GNU/Linux init.</p>Conversion Sequence Precedence in C++2018-03-17T01:00:00+08:002018-03-17T01:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-03-17:/conversion-sequence-precedence-cpp.html<h2>The story</h2>
<p>Consider the following code snippet:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><iostream></span>
<span class="k">class</span><span class="w"> </span><span class="nc">A</span><span class="p">;</span>
<span class="k">class</span><span class="w"> </span><span class="nc">B</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">public</span><span class="o">:</span><span class="w"> </span>
<span class="w"> </span><span class="n">B</span><span class="p">()</span><span class="w"> </span><span class="p">{}</span><span class="w"> </span>
<span class="w"> </span><span class="n">B</span><span class="p">(</span><span class="n">A</span><span class="w"> </span><span class="o">&</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// conversion constructor that takes cv-unqualified A</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"called B's conversion constructor"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span>
<span class="p">};</span>
<span class="k">class</span><span class="w"> </span><span class="nc">A</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">public</span><span class="o">:</span><span class="w"> </span>
<span class="w"> </span><span class="k">operator</span><span class="w"> </span><span class="n">B</span><span class="p">()</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// conversion operator that takes cv-qualified A</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"called A's conversion …</span></code></pre></div><h2>The story</h2>
<p>Consider the following code snippet:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><iostream></span>
<span class="k">class</span><span class="w"> </span><span class="nc">A</span><span class="p">;</span>
<span class="k">class</span><span class="w"> </span><span class="nc">B</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">public</span><span class="o">:</span><span class="w"> </span>
<span class="w"> </span><span class="n">B</span><span class="p">()</span><span class="w"> </span><span class="p">{}</span><span class="w"> </span>
<span class="w"> </span><span class="n">B</span><span class="p">(</span><span class="n">A</span><span class="w"> </span><span class="o">&</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// conversion constructor that takes cv-unqualified A</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"called B's conversion constructor"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span>
<span class="p">};</span>
<span class="k">class</span><span class="w"> </span><span class="nc">A</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">public</span><span class="o">:</span><span class="w"> </span>
<span class="w"> </span><span class="k">operator</span><span class="w"> </span><span class="n">B</span><span class="p">()</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// conversion operator that takes cv-qualified A</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"called A's conversion operator"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">B</span><span class="p">();</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span>
<span class="p">};</span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="n">a</span><span class="p">;</span>
<span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="n">bb</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">static_cast</span><span class="o"><</span><span class="n">B</span><span class="o">></span><span class="p">(</span><span class="n">a</span><span class="p">);</span><span class="w"> </span><span class="c1">// who gets called here? <case 1></span>
<span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">A</span><span class="p">();</span><span class="w"> </span><span class="c1">// who gets called here? <case 2></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>The output for the above code snippet is:</p>
<div class="highlight"><pre><span></span><code><span class="n">called</span><span class="w"> </span><span class="sa">B</span><span class="s1">'s conversion constructor</span>
<span class="n">called</span><span class="w"> </span><span class="n">A</span><span class="s1">'s conversion operator</span>
</code></pre></div>
<p>The exhibited behaviour completely blew me down. Why does a <strong>cast</strong> call the <em>conversion constructor</em> and an <strong>initialization</strong>
call the <em>conversion operator</em>? Isn't the expected behaviour completely the reverse? After some analysis, this became clear,
though in an unexpected way.</p>
<h2>Case 1</h2>
<p>This case is fairly simple.</p>
<p><a href="http://eel.is/c++draft/expr.static.cast#4"><code>[expr.static.cast/4]</code></a> (this is clickable; so are other specification references):</p>
<blockquote>
<p>An expression e can be explicitly converted to a type T if [...] overload resolution for a direct-initialization of an object or reference of type T from e would find at least one viable function ([over.match.viable]). If T is a reference type, the effect is the same as performing the declaration and initialization</p>
<blockquote>
<p>T t(e);</p>
</blockquote>
<p>for some invented temporary variable t ([dcl.init]) and then using the temporary variable as the result of the conversion. Otherwise, the result object is direct-initialized from e. </p>
</blockquote>
<p>Thus <code>static_cast<B>(a)</code> results in <code>A</code>'s conversion constructor getting called, regardless of presence of the conversion operator;
while if the conversion constructor <code>B::B(A &)</code> gets deleted, the following rule regarding the conversion functions apply:</p>
<p><a href="http://eel.is/c++draft/class.conv.fct"><code>[class.conv.fct]</code></a>:</p>
<blockquote>
<p>A member function of a class X having no parameters with a name of the form</p>
<blockquote>
<p>conversion-function-id:
operator conversion-type-id
conversion-type-id:
type-specifier-seq conversion-declarator_(opt)
conversion-declarator:
ptr-operator conversion-declarator_(opt)</p>
</blockquote>
<p>specifies a conversion from X to the type specified by the conversion-type-id. Such functions are called conversion functions. </p>
</blockquote>
<p>Thus <code>A::operator B()</code> gets called.</p>
<h2>Case 2</h2>
<p>This gets a little bit complicated. Note that in</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">A</span><span class="p">();</span>
</code></pre></div>
<p><code>A()</code> is an rvalue reference. And, for the purpose of overload resolution, there's an <em>implicit object parameter</em> for
<code>A::operator B()</code>, whose type is <code>cv A&</code>. This parameter is special that it can be bound to an rvalue even if it's an
lvalue reference to non-const type, according to the following:</p>
<p><a href="http://eel.is/c++draft/over.match.funcs#5"><code>[over.match.funcs/5]</code></a>:</p>
<blockquote>
<p>During overload resolution, the implied object argument is indistinguishable from other arguments. The implicit object parameter, however, retains its identity since no user-defined conversions can be applied to achieve a type match with it. For non-static member functions declared without a ref-qualifier, an additional rule applies:</p>
<blockquote>
<p>even if the implicit object parameter is not const-qualified, an rvalue can be bound to the parameter as long as in all other respects the argument can be converted to the type of the implicit object parameter. [ Note: The fact that such an argument is an rvalue does not affect the ranking of implicit conversion sequences. — end note ]</p>
</blockquote>
</blockquote>
<p>and <a href="http://eel.is/c++draft/over.match.copy#1"><code>[over.match.copy/1]</code></a>:</p>
<blockquote>
<p>Under the conditions specified in [dcl.init], as part of a copy-initialization of an object of class type, a user-defined conversion can be invoked to convert an initializer expression to the type of the object being initialized. Overload resolution is used to select the user-defined conversion to be invoked. [ Note: The conversion performed for indirect binding to a reference to a possibly cv-qualified class type is determined in terms of a corresponding non-reference copy-initialization. — end note ] Assuming that “cv1 T” is the type of the object being initialized, with T a class type, the candidate functions are selected as follows:</p>
<blockquote>
<p>The converting constructors of T are candidate functions.</p>
<p>When the type of the initializer expression is a class type “cv S”, the non-explicit conversion functions of S and its base classes are considered. When initializing a temporary object ([class.mem]) to be bound to the first parameter of a constructor where the parameter is of type “reference to possibly cv-qualified T” and the constructor is called with a single argument in the context of direct-initialization of an object of type “cv2 T”, explicit conversion functions are also considered. Those that are not hidden within S and yield a type whose cv-unqualified version is the same type as T or is a derived class thereof are candidate functions. Conversion functions that return “reference to X” return lvalues or xvalues, depending on the type of reference, of type X and are therefore considered to yield X for this process of selecting candidate functions.</p>
</blockquote>
</blockquote>
<p>together with <a href="http://eel.is/c++draft/over.ics.ref#3"><code>[over.ics.ref/3]</code></a>:</p>
<blockquote>
<p>Except for an implicit object parameter, for which see [over.match.funcs], a standard conversion sequence cannot be formed if it requires binding an lvalue reference other than a reference to a non-volatile const type to an rvalue or binding an rvalue reference to an lvalue other than a function lvalue. [ Note: This means, for example, that a candidate function cannot be a viable function if it has a non-const lvalue reference parameter (other than the implicit object parameter) and the corresponding argument would require a temporary to be created to initialize the lvalue reference (see [dcl.init.ref]). — end note ]</p>
</blockquote>
<p>So if the conversion constructor takes a non-const parameter, it is not viable; while the conversion operator is always viable,
which makes overload resolution always choose the conversion operator.</p>
<p>If we want to use temporaries for initializing via the conversion constructor, a constructor that takes an rvalue reference will
be needed, like this:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">B</span><span class="p">(</span><span class="n">A</span><span class="w"> </span><span class="o">&&</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// conversion constructor that takes cv-unqualified A as rvalue reference</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"called B's conversion constructor - rvalue reference"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span>
</code></pre></div>
<h3>Rank (precedence) between conversion sequences</h3>
<p>Another aspect to take into consideration is the rank between the conversion sequences.</p>
<p><a href="http://eel.is/c++draft/over.ics.rank#3.2"><code>[over.ics.rank/3.2]</code></a>:</p>
<blockquote>
<p>Standard conversion sequence S1 is a better conversion sequence than standard conversion sequence S2 if</p>
</blockquote>
<p><a href="http://eel.is/c++draft/over.ics.rank#3.2.6"><code>[over.ics.rank/3.2.6]</code></a>:</p>
<blockquote>
<blockquote>
<p>S1 and S2 are reference bindings, and the types to which the references refer are the same type except for top-level cv-qualifiers, and the type to which the reference initialized by S2 refers is more cv-qualified than the type to which the reference initialized by S1 refers. [ Example: ... — end example ]</p>
</blockquote>
</blockquote>
<p>So, for the rvalue reference version of conversion constructor and the conversion operator,</p>
<ul>
<li><code>B::B(A &&)</code> gets chosen instead of <code>A::operator B() const</code>,</li>
<li><code>A::operator B()</code> gets chosen instead of <code>B::B(const A &&)</code>,</li>
<li><code>B::B(A &&)</code> with <code>A::operator B()</code> / <code>B::B(const A &&)</code> with <code>A::operator B() const</code> both result in ambiguity.</li>
</ul>Meet YubiKey2018-03-13T22:00:00+08:002018-03-13T22:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-03-13:/meet-yubikey.html<h2>The story</h2>
<p>Thanks to the signing party and Harry Chen being so generous that he gave out one of his spare YubiKey 4's to me, I've
got to get my hands on this tiny new device, which really changes the game of secure authentication and encryption.
This article mainly serves …</p><h2>The story</h2>
<p>Thanks to the signing party and Harry Chen being so generous that he gave out one of his spare YubiKey 4's to me, I've
got to get my hands on this tiny new device, which really changes the game of secure authentication and encryption.
This article mainly serves as a concise introduction to the YubiKey, as well as serving as a memo in case I mess things
up (such as losing the YubiKey, breaking it or screwing up the things stored inside) and want to start all over again.</p>
<h2>What's a YubiKey?</h2>
<p>So, basically what's a <strong>YubiKey</strong>? According to Wikipedia,</p>
<blockquote>
<p>The <strong>YubiKey</strong> is a hardware authentication device manufactured by Yubico that supports one-time passwords, public key
encryption and authentication, and the Universal 2nd Factor (U2F) protocol developed by the FIDO Alliance (FIDO U2F).
It allows users to securely log into their accounts by emitting one-time passwords or using a FIDO-based public/private
key pair generated by the device. YubiKey also allows for storing static passwords for use at sites that do not support
one-time passwords.</p>
</blockquote>
<p>And I'm currently utilizing the following capabilities of a YubiKey to do things like these:</p>
<ul>
<li><strong>PIV capability</strong> for OS logon and authentication<ul>
<li>on macOS this works for both system logon / screensaver deactivation as well as sudo authentication...</li>
<li>what's more, you can ask macOS to activate the screensaver automatically when the <em>login token</em> is removed </li>
</ul>
</li>
<li><strong>U2F capability</strong> for 2FA at various websites that supports hardware U2F tokens, such as Google and GitHub</li>
<li><strong>OTP capability</strong> (Yubico OTP, to be more specific), to authenticate myself in PAM applications (e.g. sudo) on remote machines</li>
<li><strong>PGP capability</strong> for storing my OpenPGP secret keys for signing, authentication and (en|de)cryption<ul>
<li>leverage PGP in emails and live chats to <s>circumvent censorship</s> retain privacy better</li>
<li>with the help of a properly configured GPG agent to replace SSH agent, authenticate with remote machines with GPG's ssh
emulation mode</li>
</ul>
</li>
</ul>
<h2>Details</h2>
<p>Documents for how to configure each subsystem of the YubiKey are listed for reference, and I will not duplicate them here.</p>
<ul>
<li><strong>PIV</strong>: <a href="https://www.yubico.com/support/knowledge-base/categories/articles/how-to-use-your-yubikey-with-macos-sierra/">https://www.yubico.com/support/knowledge-base/categories/articles/how-to-use-your-yubikey-with-macos-sierra/</a>
to enable "login token removal = screensaver activation":<ul>
<li>go to <code>System Preferences > Security & Privacy > General > Advanced...</code> (you may need to unlock the panel by clicking on the
lock icon and authenticating)</li>
<li>check <code>Turn on screen saver when login token is removed</code></li>
</ul>
</li>
<li><strong>U2F</strong>: the simplest; follow the setup instructions from major websites that support U2F as 2FA<ul>
<li>you may need to unblock the U2F capability if you happen to turn it off (or you notice that it had stopped working):
<a href="https://wiki.archlinux.org/index.php/yubikey#Set_the_enabled_modes">https://wiki.archlinux.org/index.php/yubikey#Set_the_enabled_modes</a></li>
</ul>
</li>
<li><strong>OTP</strong>: <a href="https://mig5.net/content/adding-yubikey-2-factor-authentication-ssh-and-sudo-debian">https://mig5.net/content/adding-yubikey-2-factor-authentication-ssh-and-sudo-debian</a></li>
<li><strong>PGP</strong>: <a href="https://github.com/drduh/YubiKey-Guide">https://github.com/drduh/YubiKey-Guide</a></li>
</ul>
<h2>Credit</h2>
<p>Credit goes to Harry Chen for offering me with a free YubiKey 4, and <a href="https://tuna.moe/">Tsinghua TUNA Association</a> for their
wonderful PGP Introduction talk, as well as the PGP Signing Party they threw, where I get my first PGP key signs.</p>Dual Disk LVM with Loopback Files2018-02-05T12:00:00+08:002018-02-05T12:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-02-05:/dual-disk-lvm-with-loopback.html<h2>The story</h2>
<p>My SPARC Enterprise T5120 came with two 10000 RPM SAS HDDs, both with a labeled capacity of 146GB. Gentoo was installed on the first
hard drive, and thanks to the old SILO that was in charge of booting the system up, I have to keep a single-large-root partition …</p><h2>The story</h2>
<p>My SPARC Enterprise T5120 came with two 10000 RPM SAS HDDs, both with a labeled capacity of 146GB. Gentoo was installed on the first
hard drive, and thanks to the old SILO that was in charge of booting the system up, I have to keep a single-large-root partition
scheme. While the second disk seemed to be in some odd state (it had a block size of 528) and can't get recognized by Linux's SCSI
driver, I got it formatted with <code>sg_format</code>, turning it into a data disk with single partition on disk with ext4, and used it as the
default download location for Deluge, my BitTorrent client. So far, so good.</p>
<p>Yet 146GB is damn too tight these days, and soon it got filled up. Yet, the first disk, which was completely used as the root drive,
had around 130GB to spare. So is there a plan so that I can utilize <em>all</em> the free space on the two disks, say, to combine them into
a <em>big new pool</em>?</p>
<h2>The solution</h2>
<p>Fortunately, LVM comes to the rescue. It's trivial to create empty files, set them up as loopback devices, and make them considered pv's
by the system. Thanks to <code>truncate</code>, we can create files with big sizes, yet doesn't take up actual space.</p>
<h3>Creating the filesystem</h3>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>/
truncate<span class="w"> </span>pool_member.img<span class="w"> </span>-s<span class="w"> </span>135G<span class="w"> </span><span class="c1"># won't consume disk space</span>
losetup<span class="w"> </span>/dev/loop0<span class="w"> </span>/pool_member.img
<span class="nb">cd</span><span class="w"> </span>/var/lib/deluge<span class="w"> </span><span class="c1"># mountpoint for /dev/sdb1</span>
truncate<span class="w"> </span>pool_member.img<span class="w"> </span>-s<span class="w"> </span>135G<span class="w"> </span><span class="c1"># won't consume disk space</span>
losetup<span class="w"> </span>/dev/loop1<span class="w"> </span>/var/lib/deluge/pool_member.img
pvcreate<span class="w"> </span>/dev/loop0
pvcreate<span class="w"> </span>/dev/loop1
vgcreate<span class="w"> </span>data<span class="w"> </span>/dev/loop0<span class="w"> </span>/dev/loop1
lvcreate<span class="w"> </span>-i<span class="w"> </span><span class="m">2</span><span class="w"> </span>-l<span class="w"> </span>+100%FREE<span class="w"> </span>-n<span class="w"> </span>download<span class="w"> </span>data
mkfs.ext4<span class="w"> </span>/dev/data/download
mkdir<span class="w"> </span>-p<span class="w"> </span>/pool
mount<span class="w"> </span>/dev/data/download<span class="w"> </span>/pool
</code></pre></div>
<p>Check with <code>df -h</code> and we can see that there's a brand new filesystem mounted there, taking space from the two drives. What I had to do
next was simply moving the existing torrents over with Deluge's "Move Storage" feature.</p>
<h3>Automatically mounting the LV</h3>
<p>Wait a second. Even though the volume is up and running, it would be too much labor having to do it manually every time the system reboots.
The following init script for OpenRC works well to solve this problem:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/sbin/openrc-run</span>
<span class="c1"># Copyright 1999-2018 Gentoo Foundation</span>
<span class="c1"># Distributed under the terms of the GNU General Public License v2</span>
<span class="nv">lv_file</span><span class="o">=</span><span class="s2">"/dev/data/download"</span>
wait_file<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">file</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="nb">shift</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">wait_seconds</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">10</span><span class="si">}</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="nb">shift</span><span class="w"> </span><span class="c1"># 10 seconds as default timeout</span>
<span class="w"> </span><span class="k">until</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span><span class="k">$((</span><span class="nv">wait_seconds</span><span class="o">--</span><span class="k">))</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">0</span><span class="w"> </span>-o<span class="w"> </span>-e<span class="w"> </span><span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>sleep<span class="w"> </span><span class="m">1</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
<span class="w"> </span><span class="o">((</span>++wait_seconds<span class="o">))</span>
<span class="o">}</span>
depend<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>need<span class="w"> </span>localmount
<span class="w"> </span>before<span class="w"> </span>net
<span class="o">}</span>
start<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>ebegin<span class="w"> </span><span class="s2">"Starting mount local disk pool"</span>
<span class="w"> </span>losetup<span class="w"> </span>/dev/loop0<span class="w"> </span>/pool_member.img
<span class="w"> </span>losetup<span class="w"> </span>/dev/loop1<span class="w"> </span>/var/lib/deluge/pool_member.img
<span class="w"> </span>vgscan
<span class="w"> </span>einfo<span class="w"> </span><span class="s2">"Waiting for LV to appear..."</span>
<span class="w"> </span>wait_file<span class="w"> </span><span class="s2">"</span><span class="nv">$lv_file</span><span class="s2">"</span><span class="w"> </span><span class="m">5</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>eend<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="s2">"LV did not show up after waiting for 5 seconds"</span>
<span class="w"> </span><span class="o">}</span>
<span class="w"> </span>mount<span class="w"> </span>/dev/data/download<span class="w"> </span>/pool
<span class="w"> </span>eend<span class="w"> </span><span class="nv">$?</span><span class="w"> </span><span class="s2">"Failed to mount /pool"</span>
<span class="o">}</span>
stop<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>ebegin<span class="w"> </span><span class="s2">"Stopping mount local disk pool"</span>
<span class="w"> </span>umount<span class="w"> </span>/pool
<span class="w"> </span>vgchange<span class="w"> </span>-an<span class="w"> </span>/dev/data
<span class="w"> </span>losetup<span class="w"> </span>-d<span class="w"> </span>/dev/loop0
<span class="w"> </span>losetup<span class="w"> </span>-d<span class="w"> </span>/dev/loop1
<span class="w"> </span>eend<span class="w"> </span><span class="nv">$?</span><span class="w"> </span><span class="s2">"Failed to unmount /pool"</span>
<span class="o">}</span>
</code></pre></div>
<p>Save it to <code>/etc/init.d/disk-pool</code>, give it <code>+x</code> permission, and add it to the default runlevel with</p>
<div class="highlight"><pre><span></span><code>rc-update add disk-pool default
</code></pre></div>
<h2>Credit</h2>
<p>Credit for this method goes to lilydjwg. Thanks!</p>Go on Gentoo SPARC2018-01-22T16:00:00+08:002018-02-02T23:00:00+08:00Pengcheng Xutag:jsteward.moe,2018-01-22:/go-on-gentoo-sparc.html<h2>Preface</h2>
<p>Gc (Go's most widely-used compiler) isn't available for <code>GOOS=linux && GOARCH=sparc</code>
(Gentoo SPARC) for now; what makes matters worse is that Gccgo (<code>sys-devel/gcc[go]</code>)
is not working on Gentoo (it hangs in early initialization). Debian Ports comes to
the rescue, as although no Gc as well, their …</p><h2>Preface</h2>
<p>Gc (Go's most widely-used compiler) isn't available for <code>GOOS=linux && GOARCH=sparc</code>
(Gentoo SPARC) for now; what makes matters worse is that Gccgo (<code>sys-devel/gcc[go]</code>)
is not working on Gentoo (it hangs in early initialization). Debian Ports comes to
the rescue, as although no Gc as well, their Gccgo (at least <code>gccgo-6</code>; <code>gccgo-7</code> panics
in early stages) is working. Yet Debian's SPARC port is on architecture <code>sparc64</code>, which
means it'll run on a Gentoo kernel (UltraSPARC T2 requires a 64bit kernel), yet the
executables it creates won't run outside without some tricks. (Gentoo SPARC has 32bit userland)</p>
<p><strong>EDIT:</strong> as of Feb 2, 2018, <code>gccgo-7</code> works as well. It seems like they've fixed it. Kudos for
Debian team!</p>
<h2>Set up Debian Ports environment</h2>
<p>Install debootstrap:</p>
<div class="highlight"><pre><span></span><code>emerge<span class="w"> </span>--ask<span class="w"> </span>--verbose<span class="w"> </span>dev-util/debootstrap
</code></pre></div>
<p>Make the debian root and create the chroot with <code>debootstrap</code>:</p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>-p<span class="w"> </span>/var/debian/
debootstrap<span class="w"> </span>--arch<span class="o">=</span>sparc64<span class="w"> </span>--variant<span class="o">=</span>buildd<span class="w"> </span>--verbose<span class="w"> </span>sid<span class="w"> </span>/var/debian/<span class="w"> </span>https://deb.debian.org/debian-ports
</code></pre></div>
<p>Download the required keychain package. Chroot into the new debian environemnt, install keychain, and install required packages:</p>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>/var/debian/root
wget<span class="w"> </span>http://deb.debian.org/debian-ports/pool/main/d/debian-ports-archive-keyring/debian-ports-archive-keyring_2018.01.05_all.deb
chroot<span class="w"> </span>/var/debian<span class="w"> </span>/bin/su
chsh<span class="w"> </span>-s<span class="w"> </span>/bin/bash
dpkg<span class="w"> </span>-i<span class="w"> </span>/root/debian-ports-archive-keyring_2018.01.05_all.deb
apt<span class="w"> </span>update
apt<span class="w"> </span>install<span class="w"> </span>gccgo-6
</code></pre></div>
<p>Symlink the required go tools, make the GOPATH, and set it upon entering the chroot:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span>a<span class="w"> </span><span class="k">in</span><span class="w"> </span>go<span class="o">{</span>,fmt<span class="o">}</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>ln<span class="w"> </span>-sf<span class="w"> </span>/usr/bin/<span class="nv">$a</span>-6<span class="w"> </span>/usr/bin/<span class="nv">$a</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
mkdir<span class="w"> </span>-p<span class="w"> </span>/root/go
<span class="nb">echo</span><span class="w"> </span><span class="s2">"export GOPATH=/root/go"</span><span class="w"> </span>>><span class="w"> </span>/root/.bashrc
</code></pre></div>
<p>Exit the chroot, enter it once again, and verify that everything's working:</p>
<div class="highlight"><pre><span></span><code><span class="nb">exit</span>
chroot<span class="w"> </span>/var/debian<span class="w"> </span>/bin/su
go<span class="w"> </span>env
</code></pre></div>
<p>Test the toolchain via a simple <code>Hello, world!</code>:</p>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span>
cat<span class="w"> </span>><span class="w"> </span>helloworld.go<span class="w"> </span><span class="s"><< EOF</span>
<span class="s">package main</span>
<span class="s">import "fmt"</span>
<span class="s">func main() {</span>
<span class="s"> fmt.Println("Hello, world!")</span>
<span class="s">}</span>
<span class="s">EOF</span>
go<span class="w"> </span>run<span class="w"> </span>helloworld.go
go<span class="w"> </span>build<span class="w"> </span>helloworld.go
./helloworld
</code></pre></div>
<h2>Set up Gentoo to use the executables from Debian chroot</h2>
<p>Now that we can produce executables, we need to set up the Gentoo system to be
able to use the executables from Debian chroot. Debian has 64bit libc:</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="nv">@hikari</span><span class="err">:</span><span class="o">~</span><span class="err">#</span><span class="w"> </span><span class="n">ldd</span><span class="w"> </span><span class="n">helloworld</span>
<span class="w"> </span><span class="n">libgo</span><span class="p">.</span><span class="n">so</span><span class="mf">.9</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libgo</span><span class="p">.</span><span class="n">so</span><span class="mf">.9</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800100230000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libm</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libm</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff8001015ac000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libgcc_s</span><span class="p">.</span><span class="n">so</span><span class="mf">.1</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libgcc_s</span><span class="p">.</span><span class="n">so</span><span class="mf">.1</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800101794000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libc</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libc</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff8001018a8000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libpthread</span><span class="p">.</span><span class="n">so</span><span class="mf">.0</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libpthread</span><span class="p">.</span><span class="n">so</span><span class="mf">.0</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800101b18000</span><span class="p">)</span>
<span class="w"> </span><span class="o">/</span><span class="n">lib64</span><span class="o">/</span><span class="n">ld</span><span class="o">-</span><span class="n">linux</span><span class="p">.</span><span class="n">so</span><span class="mf">.2</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800100000000</span><span class="p">)</span>
</code></pre></div>
<p>And for <code>gccgo-7</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="nv">@hikari</span><span class="err">:</span><span class="o">~</span><span class="err">#</span><span class="w"> </span><span class="n">ldd</span><span class="w"> </span><span class="n">helloworld</span>
<span class="w"> </span><span class="n">libgo</span><span class="p">.</span><span class="n">so</span><span class="mf">.11</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libgo</span><span class="p">.</span><span class="n">so</span><span class="mf">.11</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800100230000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libm</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libm</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800101b60000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libgcc_s</span><span class="p">.</span><span class="n">so</span><span class="mf">.1</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libgcc_s</span><span class="p">.</span><span class="n">so</span><span class="mf">.1</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800101d44000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libc</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libc</span><span class="p">.</span><span class="n">so</span><span class="mf">.6</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800101e58000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libz</span><span class="p">.</span><span class="n">so</span><span class="mf">.1</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libz</span><span class="p">.</span><span class="n">so</span><span class="mf">.1</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff8001020c0000</span><span class="p">)</span>
<span class="w"> </span><span class="n">libpthread</span><span class="p">.</span><span class="n">so</span><span class="mf">.0</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">sparc64</span><span class="o">-</span><span class="n">linux</span><span class="o">-</span><span class="n">gnu</span><span class="o">/</span><span class="n">libpthread</span><span class="p">.</span><span class="n">so</span><span class="mf">.0</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff8001021dc000</span><span class="p">)</span>
<span class="w"> </span><span class="o">/</span><span class="n">lib64</span><span class="o">/</span><span class="n">ld</span><span class="o">-</span><span class="n">linux</span><span class="p">.</span><span class="n">so</span><span class="mf">.2</span><span class="w"> </span><span class="p">(</span><span class="mh">0xffff800100000000</span><span class="p">)</span>
</code></pre></div>
<p>We need to copy those to the Gentoo host's <code>/lib64</code> and add it to <code>ld</code>'s search paths.</p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>-p<span class="w"> </span>/lib64
<span class="k">for</span><span class="w"> </span>a<span class="w"> </span><span class="k">in</span><span class="w"> </span>/usr/lib/sparc64-linux-gnu/libgo.so.<span class="o">{</span><span class="m">9</span>,11<span class="o">}</span><span class="w"> </span>/lib/sparc64-linux-gnu/<span class="o">{</span>libm.so.6,libgcc_s.so.1,libc.so.6,libz.so.1,libpthread.so.0<span class="o">}</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>cp<span class="w"> </span>-Lv<span class="w"> </span>/var/debian<span class="nv">$a</span><span class="w"> </span>/lib64/<span class="k">$(</span>basename<span class="w"> </span><span class="nv">$a</span><span class="k">)</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
cp<span class="w"> </span>-v<span class="w"> </span>/var/debian/lib/sparc64-linux-gnu/ld-2.26.so<span class="w"> </span>/lib64/ld-linux.so.2
cp<span class="w"> </span>-Pv<span class="w"> </span>/var/debian/lib/sparc64-linux-gnu/libnss*<span class="w"> </span>/lib64<span class="w"> </span><span class="c1"># needed for proper user / dns support</span>
sed<span class="w"> </span>-i<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s|/lib:/usr/lib|/lib:/lib64:/usr/lib|'</span><span class="w"> </span>/etc/env.d/00basic
env-update
</code></pre></div>
<p><strong>Note:</strong> if you upgrade the debian chroot, watch closely if <code>libc</code> and friends have been updated.
If so, remember to copy the libraries over to avoid possible breakage.</p>
<p>We can now check if the executable works outside the chroot:</p>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>/var/debian/root
ldd<span class="w"> </span>helloworld
./helloworld
</code></pre></div>
<p>If any libraries are missing, we need to copy it over (to <code>/lib64</code>) and re-generate
<code>ld.so.cache</code> with <code>env-update</code>. From this point we're able to compile the Go source code in
the Debian chroot, and copy it over for use.</p>
<h2>Extra setup for ease of use</h2>
<p>Copying the executable outside the chroot outside every time is exhausting. The following setup
will ease the process of compiling source and using the executable.</p>
<p>Set up fstab entries for the debian chroot, so that it will be ready for use right after
boot. Failing to mount system psuedo-filesystems (especially <code>/proc</code>) will result in
mysterious errors with <code>libbacktrace</code>.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># debian chroot mounts</span>
<span class="o">/</span><span class="n">dev</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">debian</span><span class="o">/</span><span class="n">dev</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">defaults</span><span class="p">,</span><span class="n">rbind</span><span class="p">,</span><span class="n">rslave</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">sys</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">debian</span><span class="o">/</span><span class="n">dev</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">defaults</span><span class="p">,</span><span class="n">rbind</span><span class="p">,</span><span class="n">rslave</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="o">/</span><span class="n">proc</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">debian</span><span class="o">/</span><span class="n">proc</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">defaults</span><span class="p">,</span><span class="n">rbind</span><span class="p">,</span><span class="n">rslave</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
<span class="c1"># mount to expose the executables built to the system outside</span>
<span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">debian</span><span class="o">/</span><span class="n">root</span><span class="o">/</span><span class="n">go</span><span class="o">/</span><span class="n">bin</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">local</span><span class="o">/</span><span class="n">go</span><span class="o">/</span><span class="n">bin</span><span class="w"> </span><span class="n">none</span><span class="w"> </span><span class="n">defaults</span><span class="p">,</span><span class="n">bind</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>The last entry exposes <code>${GOPATH}/bin</code> inside the debian chroot to the system
outside, so that we can add <code>/usr/local/go/bin</code> to <code>PATH</code>, and easily use them
outside (as we have set up appropriate <code>libc</code> outside).</p>
<p>Add the <code>/usr/local/go/bin</code> to <code>PATH</code>:</p>
<div class="highlight"><pre><span></span><code><span class="nb">echo</span><span class="w"> </span><span class="s2">"PATH=\"/usr/local/go/bin\"\nROOTPATH=\"/usr/local/go/bin\""</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>tee<span class="w"> </span>/etc/env.d/40debian-golang
env-update
</code></pre></div>
<p>And add the following alias to your shell config (<code>.bashrc</code> or <code>.zshrc</code>):</p>
<div class="highlight"><pre><span></span><code><span class="nb">alias</span><span class="w"> </span><span class="nv">gobuild</span><span class="o">=</span><span class="s1">'sudo chroot /var/debian /bin/bash --login'</span>
</code></pre></div>
<p>We can now issue <code>gobuild</code> from Gentoo, go into the Debian chroot, issue</p>
<div class="highlight"><pre><span></span><code><span class="nx">go</span><span class="w"> </span><span class="nx">get</span><span class="w"> </span><span class="nx">path</span><span class="o">/</span><span class="nx">to</span><span class="o">/</span><span class="nx">your</span><span class="o">/</span><span class="nx">great</span><span class="o">/</span><span class="kn">package</span>
</code></pre></div>
<p>to install your binary, and then you can access them outside the chroot as well,
thanks to the bind mount and the <code>PATH</code> settings.</p>
<h2>Credits</h2>
<p>The method to resolve the incompatiable libc libraries comes from lilydjwg, while the suggestion
of installing the keychain package instead of importing raw keys comes from zhsj. Thanks.</p>Server naming convention2017-11-18T01:00:00+08:002017-11-18T01:00:00+08:00Pengcheng Xutag:jsteward.moe,2017-11-18:/server-naming-convention.html<h2>Preface</h2>
<p>With a bunch of servers, it's necessary that they're named in a consistent manner. This article describes how the
machines should be named.</p>
<h2>Basic rule</h2>
<p>A machine name should consist of the following three parts:</p>
<ul>
<li>a unique identifier for the machine (Romaji of Japanese words in one Kanji, examples …</li></ul><h2>Preface</h2>
<p>With a bunch of servers, it's necessary that they're named in a consistent manner. This article describes how the
machines should be named.</p>
<h2>Basic rule</h2>
<p>A machine name should consist of the following three parts:</p>
<ul>
<li>a unique identifier for the machine (Romaji of Japanese words in one Kanji, examples are given below)</li>
<li>the name of the ISP (e.g. <code>vultr</code> or <code>linode</code>)</li>
<li>the IATA code of the geographically closest airport (e.g. <code>hnd</code> for machines in Tokyo)</li>
</ul>
<h2>Examples for the Japanese words to be chosen</h2>
<div class="highlight"><pre><span></span><code>光(hikari) 影(kage) 日(hi) 月(tsuki) 雨(ame) 雪(yuki) 風(kaze) 嵐(arashi)
海(umi) 川(kawa) 山(yama) 空(sora) 森(mori) 池(ike) 滝(taki) 島(shima)
関(seki) 原(hara) 牧(maki) 崎(saki) 谷(tani) 岩(iwa) 垣(kaki) 塵(chiri)
岳(take) 稲(ina) 天(ama) 埴(hani) 堀(hori) 浜(hama) 浦(ura) 沢(sawa)
</code></pre></div>
<h2>DNS rule for machines</h2>
<p>Every machine should have A and AAAA (if any) records corresponding their <strong>full</strong> name under the <code>jsteward.moe</code> domain, as well as CNAME records for
their unique identifier for ease of access. The <code>dig</code> result for <code>hikari.pku.pek</code> is listed below as an example.</p>
<div class="highlight"><pre><span></span><code><span class="o">;</span><span class="w"> </span><span class="o"><<>></span><span class="w"> </span><span class="nt">DiG</span><span class="w"> </span><span class="nt">9</span><span class="p">.</span><span class="nc">11</span><span class="p">.</span><span class="nc">2</span><span class="w"> </span><span class="o"><<>></span><span class="w"> </span><span class="nt">hikari</span><span class="p">.</span><span class="nc">jsteward</span><span class="p">.</span><span class="nc">moe</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">global</span><span class="w"> </span><span class="nt">options</span><span class="o">:</span><span class="w"> </span><span class="o">+</span><span class="nt">cmd</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">Got</span><span class="w"> </span><span class="nt">answer</span><span class="o">:</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">-</span><span class="o">>></span><span class="nt">HEADER</span><span class="o"><<</span><span class="nt">-</span><span class="w"> </span><span class="nt">opcode</span><span class="o">:</span><span class="w"> </span><span class="nt">QUERY</span><span class="o">,</span><span class="w"> </span><span class="nt">status</span><span class="o">:</span><span class="w"> </span><span class="nt">NOERROR</span><span class="o">,</span><span class="w"> </span><span class="nt">id</span><span class="o">:</span><span class="w"> </span><span class="nt">48767</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">flags</span><span class="o">:</span><span class="w"> </span><span class="nt">qr</span><span class="w"> </span><span class="nt">rd</span><span class="w"> </span><span class="nt">ra</span><span class="w"> </span><span class="nt">ad</span><span class="o">;</span><span class="w"> </span><span class="nt">QUERY</span><span class="o">:</span><span class="w"> </span><span class="nt">1</span><span class="o">,</span><span class="w"> </span><span class="nt">ANSWER</span><span class="o">:</span><span class="w"> </span><span class="nt">2</span><span class="o">,</span><span class="w"> </span><span class="nt">AUTHORITY</span><span class="o">:</span><span class="w"> </span><span class="nt">0</span><span class="o">,</span><span class="w"> </span><span class="nt">ADDITIONAL</span><span class="o">:</span><span class="w"> </span><span class="nt">1</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">OPT</span><span class="w"> </span><span class="nt">PSEUDOSECTION</span><span class="o">:</span>
<span class="o">;</span><span class="w"> </span><span class="nt">EDNS</span><span class="o">:</span><span class="w"> </span><span class="nt">version</span><span class="o">:</span><span class="w"> </span><span class="nt">0</span><span class="o">,</span><span class="w"> </span><span class="nt">flags</span><span class="o">:;</span><span class="w"> </span><span class="nt">udp</span><span class="o">:</span><span class="w"> </span><span class="nt">4096</span>
<span class="o">;</span><span class="w"> </span><span class="nt">COOKIE</span><span class="o">:</span><span class="w"> </span><span class="nt">6098f5d0466d72b4a19210815a0fbc9fecbc66d47b912c6a</span><span class="w"> </span><span class="o">(</span><span class="nt">good</span><span class="o">)</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">QUESTION</span><span class="w"> </span><span class="nt">SECTION</span><span class="o">:</span>
<span class="o">;</span><span class="nt">hikari</span><span class="p">.</span><span class="nc">jsteward</span><span class="p">.</span><span class="nc">moe</span><span class="o">.</span><span class="w"> </span><span class="nt">IN</span><span class="w"> </span><span class="nt">A</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">ANSWER</span><span class="w"> </span><span class="nt">SECTION</span><span class="o">:</span>
<span class="nt">hikari</span><span class="p">.</span><span class="nc">JSTEWARD</span><span class="p">.</span><span class="nc">moe</span><span class="o">.</span><span class="w"> </span><span class="nt">185</span><span class="w"> </span><span class="nt">IN</span><span class="w"> </span><span class="nt">CNAME</span><span class="w"> </span><span class="nt">hikari</span><span class="p">.</span><span class="nc">pku</span><span class="p">.</span><span class="nc">pek</span><span class="p">.</span><span class="nc">jsteward</span><span class="p">.</span><span class="nc">moe</span><span class="o">.</span>
<span class="nt">hikari</span><span class="p">.</span><span class="nc">pku</span><span class="p">.</span><span class="nc">pek</span><span class="p">.</span><span class="nc">JSTEWARD</span><span class="p">.</span><span class="nc">moe</span><span class="o">.</span><span class="w"> </span><span class="nt">185</span><span class="w"> </span><span class="nt">IN</span><span class="w"> </span><span class="nt">A</span><span class="w"> </span><span class="nt">222</span><span class="p">.</span><span class="nc">29</span><span class="p">.</span><span class="nc">47</span><span class="p">.</span><span class="nc">177</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">Query</span><span class="w"> </span><span class="nt">time</span><span class="o">:</span><span class="w"> </span><span class="nt">2</span><span class="w"> </span><span class="nt">msec</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">SERVER</span><span class="o">:</span><span class="w"> </span><span class="nt">101</span><span class="p">.</span><span class="nc">6</span><span class="p">.</span><span class="nc">6</span><span class="p">.</span><span class="nc">6</span><span class="p">#</span><span class="nn">53</span><span class="o">(</span><span class="nt">101</span><span class="p">.</span><span class="nc">6</span><span class="p">.</span><span class="nc">6</span><span class="p">.</span><span class="nc">6</span><span class="o">)</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">WHEN</span><span class="o">:</span><span class="w"> </span><span class="nt">土</span><span class="w"> </span><span class="nt">11月</span><span class="w"> </span><span class="nt">18</span><span class="w"> </span><span class="nt">12</span><span class="p">:</span><span class="nd">52</span><span class="p">:</span><span class="nd">46</span><span class="w"> </span><span class="nt">CST</span><span class="w"> </span><span class="nt">2017</span>
<span class="o">;;</span><span class="w"> </span><span class="nt">MSG</span><span class="w"> </span><span class="nt">SIZE</span><span class="w"> </span><span class="nt">rcvd</span><span class="o">:</span><span class="w"> </span><span class="nt">152</span>
</code></pre></div>
<p>Application domain names (e.g. <code>archive.jsteward.moe</code>) should be pointed to the short CNAME record (e.g. <code>hikari.jsteward.moe</code>).</p>MPLS in GRE tunnel on Linux with iproute22017-11-15T18:00:00+08:002018-05-16T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2017-11-15:/mpls-in-gre-tunnel-linux.html<h2>Preface</h2>
<p><a href="https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching">MPLS</a> can be used for prioritizing traffic, which will be used in JSteward Tech's routing policy.
This article works out how to label packets with MPLS and switch according to packets between two host connected by a GRE tunnel.</p>
<h2>Before you start</h2>
<p>Full MPLS support wasn't present in Linux …</p><h2>Preface</h2>
<p><a href="https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching">MPLS</a> can be used for prioritizing traffic, which will be used in JSteward Tech's routing policy.
This article works out how to label packets with MPLS and switch according to packets between two host connected by a GRE tunnel.</p>
<h2>Before you start</h2>
<p>Full MPLS support wasn't present in Linux kernel until lately (4.3 at least), so check your kernel version before trying out things in this article; incompatiable kernels will most likely give out the following two types of errors:</p>
<div class="highlight"><pre><span></span><code># ip -f mpls route add 101 dev lo
RTNETLINK answers: Operation not supported
</code></pre></div>
<p>... which says by itself pretty much. Another type of error looks like this:</p>
<div class="highlight"><pre><span></span><code># sysctl -w net.mpls.conf.foo4.input=1
sysctl: cannot stat /proc/sys/net/mpls/conf/foo4/input: No such file or directory
</code></pre></div>
<p>This means that you haven't loaded the correct kernel modules (<code>mpls_router</code> in this case). Userspace tools should get the modules loaded on demand, though; the (most likely non-exhaustive) list for MPLS-related kernel modules are as follows:</p>
<ul>
<li>mpls_router</li>
<li>mpls_gso</li>
<li>mpls_iptunnel</li>
</ul>
<p>Make sure that you have the suitable kernel as well as the correct set of kernel modules--maybe your Enterprise Linux vendor has back-ported things to older kernels, but I wouldn't be able to help in those cases: contact your vendor for support as you pay them for that. If you use Gentoo (and don't use <code>genkernel all</code>, which is evil), try enabling everything related to MPLS to avoid cryptic errors. To tell if your kernel really supports MPLS, issue the following, and you should see output like this for <strong>all</strong> of your interfaces:</p>
<div class="highlight"><pre><span></span><code><span class="gh">#</span> sysctl -a --pattern mpls
net.mpls.conf.eth0.input = 0
net.mpls.conf.eth1.input = 0
net.mpls.conf.lo.input = 0
net.mpls.platform_labels = 0
</code></pre></div>
<h2>Step 1: the GRE tunnel</h2>
<p>Let there be two machines: A and B. A has the address <code>10.128.0.2</code> and B has the address <code>10.128.100.2</code>. The following creates a GRE tunnel between the host.
We'll use the interface name <code>foo4</code> on both ends. Execute all the following with root:</p>
<p>On A:</p>
<div class="highlight"><pre><span></span><code><span class="nx">ip</span><span class="w"> </span><span class="nx">tun</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="nx">foo4</span><span class="w"> </span><span class="nx">mode</span><span class="w"> </span><span class="nx">gre</span><span class="w"> </span><span class="nx">remote</span><span class="w"> </span><span class="m m-Double">10.128.100.2</span><span class="w"> </span><span class="nx">local</span><span class="w"> </span><span class="m m-Double">10.128.0.2</span>
<span class="nx">ip</span><span class="w"> </span><span class="kd">addr</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="m m-Double">10.199.0.1</span><span class="w"> </span><span class="nx">dev</span><span class="w"> </span><span class="nx">foo4</span>
<span class="nx">ip</span><span class="w"> </span><span class="nx">link</span><span class="w"> </span><span class="nx">set</span><span class="w"> </span><span class="nx">foo4</span><span class="w"> </span><span class="nx">up</span>
<span class="nx">ip</span><span class="w"> </span><span class="nx">rou</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="m m-Double">10.199.0.0</span><span class="o">/</span><span class="mi">24</span><span class="w"> </span><span class="nx">dev</span><span class="w"> </span><span class="nx">foo4</span>
</code></pre></div>
<p>On B:</p>
<div class="highlight"><pre><span></span><code><span class="nx">ip</span><span class="w"> </span><span class="nx">tun</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="nx">foo4</span><span class="w"> </span><span class="nx">mode</span><span class="w"> </span><span class="nx">gre</span><span class="w"> </span><span class="nx">remote</span><span class="w"> </span><span class="m m-Double">10.128.0.2</span><span class="w"> </span><span class="nx">local</span><span class="w"> </span><span class="m m-Double">10.128.100.2</span>
<span class="nx">ip</span><span class="w"> </span><span class="kd">addr</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="m m-Double">10.199.0.2</span><span class="w"> </span><span class="nx">dev</span><span class="w"> </span><span class="nx">foo4</span>
<span class="nx">ip</span><span class="w"> </span><span class="nx">link</span><span class="w"> </span><span class="nx">set</span><span class="w"> </span><span class="nx">foo4</span><span class="w"> </span><span class="nx">up</span>
<span class="nx">ip</span><span class="w"> </span><span class="nx">rou</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="m m-Double">10.199.0.0</span><span class="o">/</span><span class="mi">24</span><span class="w"> </span><span class="nx">dev</span><span class="w"> </span><span class="nx">foo4</span>
</code></pre></div>
<p>Verify that the hosts can reach each other by pinging the <code>10.199.0.0/24</code> addresses.</p>
<h2>Step 2: MPLS encapsulation</h2>
<p>To encapsulate packets that go through the GRE tunnel with MPLS header, we have to replace the default route with encapsulation rules, and add corresponding
decapsulation rules on the other end. Label 100 will be used for traffic that goes from A towards B, while label 101 will be used for traffic that goes from
B towards A. Execute all the following with root:</p>
<p>On A:</p>
<div class="highlight"><pre><span></span><code>sysctl -w net.mpls.platform_labels=65535
sysctl -w net.mpls.conf.foo4.input=1
ip rou c 10.199.0.0/24 encap mpls 100 dev foo4
ip -f mpls rou a 101 dev lo
</code></pre></div>
<p>On B:</p>
<div class="highlight"><pre><span></span><code>sysctl -w net.mpls.platform_labels=65535
sysctl -w net.mpls.conf.foo4.input=1
ip rou c 10.199.0.0/24 encap mpls 101 dev foo4
ip -f mpls rou a 100 dev lo
</code></pre></div>
<p>Packets sent from one peer will be labeled with its corresponding label, and then decapsulated at the other end and delivered to its local interface.</p>
<p>Note that <code>rp_filter</code> needs to be set to relaxed mode or completely disabled for packets to be accepted correctly. Do the following on both machines:</p>
<div class="highlight"><pre><span></span><code>sysctl -w net.ipv4.conf.all.rp_filter=2
</code></pre></div>
<p>Now the both machines should be able to ping each other, and when capturing the packets with <code>tcpdump</code>, one will see that the packets have the MPLS header.</p>JSteward Tech Internal Network Design2017-11-11T15:00:00+08:002017-11-11T15:00:00+08:00Pengcheng Xutag:jsteward.moe,2017-11-11:/jsteward-tech-internal-network-design.html<h2>Preface</h2>
<p>As the number of machines I manage increase gradually, it has become a necessity to connect the machines
together through secure, reliable tunnels, thus forming an internal network. A private network can have the following
advantages:</p>
<ul>
<li>Ease of configuring services that involve multiple machines, such as a databases and …</li></ul><h2>Preface</h2>
<p>As the number of machines I manage increase gradually, it has become a necessity to connect the machines
together through secure, reliable tunnels, thus forming an internal network. A private network can have the following
advantages:</p>
<ul>
<li>Ease of configuring services that involve multiple machines, such as a databases and database clients</li>
<li>Data inside the network are controlled, thus eliminating the need of unnecessary application layer
security</li>
<li>Convenience when configuring optimal route for network clients that wish their traffic be sent through the network</li>
</ul>
<p>This article mainly describes the structure of the network (i.e. how the sites are connected), address allocation and
route selection. It should serve as the document to how the network should be implemented afterwards, though further
revisions may take place.</p>
<h2>Part one: how are machines interconnected</h2>
<p>A node in the network shall have one and only one of the following roles:</p>
<ul>
<li>a server (a.k.a. a <em>core router</em>)</li>
<li>a client (a.k.a. an <em>end user</em>)</li>
<li>a small router</li>
</ul>
<p>A <strong>server</strong> connects to other servers, forming a full mesh network between the servers. A <strong>small router</strong> behaves just
like a server, except that it only connects to <strong>one</strong> server, instead of <em>every</em> server (that's what <em>full mesh</em> means).</p>
<p>A <strong>client</strong> connects to the network via one of the following:</p>
<ul>
<li>a <strong>server</strong> (through <em>remote access</em>: using a normal VPN protocol)</li>
<li>a <strong>small router</strong> (through normal IP)</li>
</ul>
<p>The following sections define the detailed properties of possible connections between the networks.</p>
<h3><strong>Server</strong> to <strong>server</strong> and <strong>small router</strong> to <strong>server</strong></h3>
<p>The two hosts should connect to each other through GRE over IPsec. The IPsec tunnel should be managed by <code>strongswan</code>.
Both ends should authenticate each other through X.509 certificates, issued by the PKI managed by JSteward Tech.</p>
<h3><strong>Client</strong> to <strong>server</strong></h3>
<p>The client should establish a VPN session to the server, with one of the following protocols:</p>
<ul>
<li>IKEv2</li>
<li>IPsec Hybrid RSA</li>
</ul>
<h3><strong>Client</strong> to <strong>small router</strong></h3>
<p>The client should connect to the small router directly via physical media, such as an ethernet cable or a WiFi connection.</p>
<h2>Part two: how are addresses allocated</h2>
<p>JSteward Tech does not process any IP's for now, neither IPv4 nor IPv6. For IPv4 and IPv6 the policy for assigning addresses
are a little different. They're described in detail in the sections below.</p>
<h3>IPv4</h3>
<p>Judging from the current situations, it's not possible in the near future for JSteward Tech to acquire public IPv4 prefixes,
thus the private network <code>10.192.0.0/10</code> is used. All addresses will be assigned inside this network.</p>
<p>For each server, including small routers, a <code>/16</code> in the network will be assigned to it, in a continuous manner, that is, the
prefixes are assigned to each server <em>one-by-one</em>.</p>
<p>All clients that connect to a server, including small routers, will get their address from the first <code>/17</code> in the server's <code>/16</code>
network. The second <code>/17</code> is reserved for the server's own applications (such as containers).</p>
<h3>IPv6</h3>
<p>Currently, JSteward Tech doesn't possess public IPv6 addresses. Yet many of the servers, including small routers, have native
IPv6 (provided by CERNET2 or VPS provider). Such addresses are obtained via router advertisement with prefix <code>/64</code>.</p>
<p>For small routers without native IPv6, it should receive a <code>/80</code> from its upstream (i.e. the server it connects to). For servers
whose prefix is not routable (thus impossible to use <code>ndppd</code> to further split the network) or without native IPv6, it should get
its <code>/64</code> from HE tunnel broker, which should be connected to one of the servers.</p>
<p>All clients get a <code>/96</code> from its upstream, regardless of it being a small router or a server. This should be achieved through
<code>ndppd</code>, in order to get the correct routes, as prefixes longer than <code>/64</code> will break SLAAC.</p>
<p>If JSteward Tech should possess any public IP addresses, they should be assigned to servers without native IPv6, replacing what
HE tunnelbroker does for now. Due to this possibility, the IPv6 address assignment section is prone to changes.</p>
<h2>Part three: how are routes selected</h2>
<p>Note: IPv6 routes are not discussed here. All IPv6 data should simply follow its upstream's native IPv6 rules and go out into the public
Internet wherever possible.</p>
<p>The IPv4 traffic that goes through this network fall into <em>exactly</em> one of the following categories:</p>
<ul>
<li>it goes to a host <strong>inside</strong> the network (e.g. a server in the network that serves contents)</li>
<li>it goes to a host <strong>outside</strong> the network (e.g. when a client uses the network as some kind of proxy)</li>
</ul>
<p>The following sections discuss the two kinds of network traffic in detail.</p>
<h3>To a host <strong>inside</strong> the network</h3>
<p>JSteward Tech will use iBGP for route exchanging within the network. Route selection will be based on link latency and bandwidth,
with latency as the first consideration. This will be achieved through dynamically alternating the <code>bgp_community</code> values, whose data
is acquired through a live monitoring system. The following documents should help when implementing the system, thus listed here for
future reference:</p>
<ul>
<li><a href="https://dn42.eu/howto/Bird">DN42's Bird howto</a></li>
<li><a href="https://dn42.eu/howto/Bird-communities">DN42's Bird BGP community howto</a></li>
<li><a href="http://bird.network.cz/?get_doc&f=bird-6.html#ss6.3">Bird documentation</a></li>
<li><a href="https://lkhill.com/using-influxdb-grafana-to-display-network-statistics/">Example on monitoring a network and visualizing data with Grafana</a></li>
</ul>
<h3>To a host <strong>outside</strong> the network</h3>
<p>MPLS shall be utilized to identify the exit preferred when a data packet enters the core, in which the exit is selected according to
a predefined table describing which exit a packet should choose based on its destination address. The packet is then encapsulated with
the appropriate MPLS label according to the exit chosen, and routed through the core. Actions in the <a href="https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching#Label-switched_path">LSP</a>
are defined as follows:</p>
<ul>
<li>a LER (label edge router) encapsulates the packet with MPLS label selected by destination address and sends it to the next hop determined
by the optimal next hop to the egress router according to the iBGP routing table</li>
<li>a LSR (label switching router) swaps the the label with one identical to the original one (i.e. leave the label <em>untouched</em>) and sends it
to the next hop determined by optimal next hop to the egress router according to the iBGP routing table</li>
<li>the egress router decapsulates the MPLS label, does MASQUERADE and sends the packet out into the Internet.</li>
</ul>
<p>Due to the nature of needing to operate on the MPLS table every time the link latency changes (and that the iBGP routing table changes), a
helper program will be needed to update the MPLS routes every time such change occurs.</p>
<p>For a host that wishes to select the exit node of its traffic on its own instead of conforming to the predefined table at the LER, the server
can act as an LER itself, encapsulating the packets with the desired label corresponding to the exit chosen. Then the packets will be routed to
the desired exit by the LSR's on the LSP.</p>
<p>The following documents should help when implementing the system, thus listed here for future reference:</p>
<ul>
<li><a href="/mpls-in-gre-tunnel-linux.html">MPLS over GRE experiment on my own blog</a></li>
<li><a href="http://www.samrussell.nz/2015/12/mpls-testbed-on-ubuntu-linux-with.html">MPLS testbed article</a></li>
</ul>Proper routing for HE Tunnelbroker on machines with native (and preferred) IPv6 on Gentoo (with `netifrc`)2017-11-10T00:00:00+08:002017-11-10T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2017-11-10:/he-ipv6-routing-on-machines-with-ipv6.html<h2>Preface</h2>
<p>On a machine with native IPv6 that's working quite well (e.g. CERNET2),
it's usually desirable to set the host's default IPv6 route to go through
native IPv6. Yet the cheap <code>/48</code> address block offered by HE Tunnelbroker
can be useful if one wishes to assign IPv6 addresses through …</p><h2>Preface</h2>
<p>On a machine with native IPv6 that's working quite well (e.g. CERNET2),
it's usually desirable to set the host's default IPv6 route to go through
native IPv6. Yet the cheap <code>/48</code> address block offered by HE Tunnelbroker
can be useful if one wishes to assign IPv6 addresses through VPN. This
article mainly serves as a memo for achieving the following goals:</p>
<ul>
<li>native IPv6 should work just fine (i.e. <code>default</code> route untouched)</li>
<li>tunnelbroker's addresses should work as well (e.g. services that bind to the tunnelbroker's addresses should get correct routes)</li>
</ul>
<h2>The <code>default</code> route</h2>
<p>For native IPv6, a <code>default</code> route would normally look like this:</p>
<div class="highlight"><pre><span></span><code>default via fe80::c256:27ff:fed2:90fd dev wlp3s0 proto static metric 600 pref medium
</code></pre></div>
<p>... which sets the router's link-local address (acquired via router advertisement)
as the default gateway, thus defining route for all outgoing packets to go through the
physical port.</p>
<p>The problem is, the native IPv6 routers usually filter the packets based on src address ACL.
As a result, the packets with the tunnelbroker's address as the src address usually get dropped
instead of further routed, thus making the tunnel effectively useless.</p>
<p>Linux's iproute2 tool has the <code>ip-rule</code> tool, which handles policy routing, and can solve the
problem. What we need is a new <code>default</code> route for packets with src addresses that fall in
the prefix allocated from tunnelbroker. The following works like a charm:</p>
<div class="highlight"><pre><span></span><code>ip -6 route add default dev he-ipv6 table 101
ip -6 rule add from 2001:470:f838::/48 lookup 101 priority 500
</code></pre></div>
<p>... where <code>he-ipv6</code> and the prefix correspond to what you get from HE.</p>
<p>To get things fixed and automatically set up across reboots, we'll need to configure the network
manager to automatically add the RPDB rules as well as the extra routes in the special table <code>101</code>.
On Gentoo this is handled via <code>netifrc</code>, whose configuration lies in <code>/etc/conf.d/net</code>. Things should
look like this:</p>
<div class="highlight"><pre><span></span><code>rules6_he_ipv6="
from 2001:470:f838::/48 lookup he priority 500
"
routes_he_ipv6="
::/0 dev he-ipv6 table he
"
</code></pre></div>
<p>The <code>table he</code> is defined in <code>/etc/iproute2/rt_tables</code> as an alias of <code>table 101</code> so that we can get
clearer (thus better memorable) rules and routes. Note that <code>::/0</code> is used here as I can't find a way
to separate IPv4 and IPv6 routes here (!), and the simple <code>default</code> will result in the route got added
into IPv4 routing table.</p>
<h2>One more thing</h2>
<p>One thing that bothered me for quite some time is somehow unrelated to routing itself, but worth noting
as failing to notice this will result in strange results. <strong>Make sure your IPv4 firewall permits <code>ipv6</code>
packets (a.k.a. protocol 41)!</strong> Like this:</p>
<div class="highlight"><pre><span></span><code>iptables -A INPUT -p ipv6 -j ACCEPT
</code></pre></div>
<p>I used to have things like this:</p>
<div class="highlight"><pre><span></span><code>iptables -P INPUT DROP
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -P OUTPUT ACCEPT
</code></pre></div>
<p>... which results in incoming packets won't reach the destination <em>unless</em> the host sends a packet first,
thus triggering the connmark. Inbound connections fail after the tunnel turns inactive for a while, when
the conntrack expires.</p>Upgrade Gentoo FreeBSD 10.3 to 11.02017-09-07T00:00:00+08:002017-09-07T00:00:00+08:00Pengcheng Xutag:jsteward.moe,2017-09-07:/gfbsd-10.3-to-11.0.html<h2>Preface</h2>
<p>Now that we have successfully upgraded Gentoo FreeBSD from 9.1 to 10.3, we
need to upgrade it to 11.0 in order to catch up the latest FreeBSD system.
The major steps consist of the following:</p>
<ul>
<li>Upgrading world</li>
<li>Changing to the new profile</li>
<li>Upgrade the kernel</li>
<li>Upgrade …</li></ul><h2>Preface</h2>
<p>Now that we have successfully upgraded Gentoo FreeBSD from 9.1 to 10.3, we
need to upgrade it to 11.0 in order to catch up the latest FreeBSD system.
The major steps consist of the following:</p>
<ul>
<li>Upgrading world</li>
<li>Changing to the new profile</li>
<li>Upgrade the kernel</li>
<li>Upgrade the userland</li>
<li>Change <code>CHOST</code> and rebuild the toolchain</li>
</ul>
<p>Let's start.</p>
<h2>World upgrade</h2>
<p>Remember to sync the repositories beforehand to make sure we're on the most
recent version of the Portage tree.</p>
<div class="highlight"><pre><span></span><code><span class="n">emerge</span><span class="w"> </span><span class="o">--</span><span class="n">sync</span>
<span class="n">emerge</span><span class="w"> </span><span class="o">-</span><span class="n">avuDN</span><span class="w"> </span><span class="nv">@world</span>
</code></pre></div>
<h2>Switch to the latest profile</h2>
<div class="highlight"><pre><span></span><code>eselect profile list
eselect profile set default/bsd/fbsd/amd64/11.0
</code></pre></div>
<h2>Upgrade the kernel</h2>
<p>It's important to upgrade the kernel first as some userland utilities may
require the newer functions in the kernel.</p>
<div class="highlight"><pre><span></span><code>emerge -a1 --nodeps sys-freebsd/freebsd-mk-defs
emerge -a1 --nodeps sys-freebsd/freebsd-sources
reboot
</code></pre></div>
<p>Check the running kernel by <code>uname</code>:</p>
<div class="highlight"><pre><span></span><code>uname -a
</code></pre></div>
<h2>Upgrade the FreeBSD userland</h2>
<p>Check and remove strange flags from <code>CFLAGS</code>. The most basic configuration
should consist of simply <code>CFLAGS="-O2 -pipe"</code>.</p>
<div class="highlight"><pre><span></span><code>vim /etc/portage/make.conf
</code></pre></div>
<p>Seems like GCC 6 doesn't play well when building <code>freebsd-lib</code>
(<a href="https://stackoverflow.com/questions/46129786/isystem-generated-by-make-disturbing-header-search-order-with-gcc-6">here if you're interested</a>), and we'll need
<code>sys-devel/gcc:5.4.0</code> for now.</p>
<div class="highlight"><pre><span></span><code>emerge -av sys-devel/gcc:5.4.0
gcc-config -l
gcc-config 1
</code></pre></div>
<p>Upgrade the core libraries first.</p>
<div class="highlight"><pre><span></span><code>emerge -a1 --nodeps sys-freebsd/freebsd-share sys-freebsd/freebsd-lib sys-freebsd/freebsd-libexec
</code></pre></div>
<p>Upgrade the rest of the userland.</p>
<div class="highlight"><pre><span></span><code>emerge -a1uN boot0 freebsd-bin freebsd-lib freebsd-libexec freebsd-mk-defs freebsd-pam-modules freebsd-sbin freebsd-share freebsd-ubin freebsd-usbin
</code></pre></div>
<p>Re-merge all these packages once again as some of them require header files
from 11.0, which aren't available before.</p>
<div class="highlight"><pre><span></span><code>emerge -a1 boot0 freebsd-bin freebsd-lib freebsd-libexec freebsd-mk-defs freebsd-pam-modules freebsd-sbin freebsd-share freebsd-ubin freebsd-usbin
</code></pre></div>
<h2>Change the <code>CHOST</code> variable and rebuild the toolchain</h2>
<p>Edit <code>/etc/portage/make.conf</code> and change the <code>CHOST</code> variable from 10.3 to
11.0.</p>
<div class="highlight"><pre><span></span><code>vim /etc/portage/make.conf
emerge -av1 binutils gcc
</code></pre></div>
<p>Make sure you use the correct version of <code>binutils</code> and <code>gcc</code>:</p>
<div class="highlight"><pre><span></span><code>gcc-config -c
binutils-config -c
</code></pre></div>
<p>Rebuild world with the new toolchain:</p>
<div class="highlight"><pre><span></span><code><span class="n">emerge</span><span class="w"> </span><span class="o">-</span><span class="n">a1</span><span class="w"> </span><span class="n">sys</span><span class="o">-</span><span class="n">devel</span><span class="o">/</span><span class="n">libtool</span>
<span class="n">emerge</span><span class="w"> </span><span class="o">-</span><span class="n">ae</span><span class="w"> </span><span class="err">@</span><span class="n">world</span><span class="w"> </span><span class="o">--</span><span class="n">exclude</span><span class="w"> </span><span class="n">sys</span><span class="o">-</span><span class="n">apps</span><span class="o">/</span><span class="n">portage</span>
<span class="n">emerge</span><span class="w"> </span><span class="o">-</span><span class="n">a1</span><span class="w"> </span><span class="n">sys</span><span class="o">-</span><span class="n">apps</span><span class="o">/</span><span class="n">portage</span>
</code></pre></div>
<p>If any of the packages fail to build you can resume the build, skipping
that package with:</p>
<div class="highlight"><pre><span></span><code>emerge -ar --skipfirst
</code></pre></div>
<h2>Clean up</h2>
<p>Rebuild packages with preserved libraries:</p>
<div class="highlight"><pre><span></span><code><span class="n">emerge</span><span class="w"> </span><span class="nv">@preserved</span><span class="o">-</span><span class="n">rebuild</span>
<span class="n">dispatch</span><span class="o">-</span><span class="n">conf</span>
</code></pre></div>Test after migrating the server to Vultr2016-12-20T13:00:00+08:002016-12-20T13:00:00+08:00Pengcheng Xutag:jsteward.moe,2016-12-20:/server-migration-test.html<p class="first last">test if everything's correct</p>
<p>After migrating the server to Vultr and changing system to Arch Linux, I feel it necessary to perform some tests.</p>
<p>If you can read this normally and the corresponding repositor's CI hasn't failed, then everything's fine.</p>
Hello, world!2016-09-17T15:00:00+08:002016-09-17T15:00:00+08:00Pengcheng Xutag:jsteward.moe,2016-09-17:/hello-world.html<p class="first last">first run for pelican</p>
<p>My first try of static blog pages using <tt class="docutils literal">pelican</tt>.</p>
<p>No one will be reading this, right? But I'll try my best to write though..</p>