GNU Tools @ LPC Tuesday 25 Aug 2020: chat record ================================================ All times are BST (UTC + 1) Pre-start --------- [14:02] Jose E. Marchesi : Hi Sarah, Frank [14:02] Sarah Cook : Hello :) [14:02] Jose E. Marchesi : I collected all the pdfs from indico corresponding to the sessions I am moderating [14:02] Frank Ch. Eigler : Hi guys, was just about to upload the day's slides too :) [14:02] Jose E. Marchesi : One is missing though, "Linking LTO and make" from John Ravi [14:03] Jose E. Marchesi : @Frank let me know when you are done so I can take presenter and upload mines [14:03] Frank Ch. Eigler : Sarah has the conch at the moment [14:03] Jose E. Marchesi : ok [14:04] Sarah Cook : I think I have uploaded all the talks for today minus John Ravi's. Happy for you both to double check [14:05] Frank Ch. Eigler : hmmmm not sure how to see the list of uploaded presentations without becoming a presenter [14:05] Sarah Cook : I have made you the presenter [14:05] Sarah Cook : if you click the blue cross in the bottom left and "upload presentations" it will bring the list up [14:05] Jose E. Marchesi : but only if you are presenter :) [14:06] Frank Ch. Eigler : thanks, looks good, passing to Jose [14:07] Jose E. Marchesi : got it, uploading... [14:07] Sarah Cook : If you are both happy, at the end of each talk I can change over the presentations so the correct one is up ready and I can do the count downs as well? I just want to make it as simple for everyone as i can [14:08] Jose E. Marchesi : Oh you already did it [14:08] Jose E. Marchesi : only the missing lighting talk (liking lto and make) is missing now [14:08] Jose E. Marchesi : @Sarah sure [14:09] Frank Ch. Eigler : sure [14:09] Jose E. Marchesi : but then whats up to Frank and me to do? 8-) [14:09] Frank Ch. Eigler : I am GOOD at thumb twiddling, might set speed records [14:11] Sarah Cook : if you could introduce the speakers and keep questions flowing and conversations going in BoFs? [14:11] Jose E. Marchesi : ok [14:12] Jose E. Marchesi : @Sarah but you will be giving/keeping presenter right? [14:13] Sarah Cook : Yes, I will try to [14:13] Jose E. Marchesi : ok [14:23] Simon Marchi : Good morning (or other) [14:23] Jose E. Marchesi : hi [14:23] Sarah Cook : hello [14:25] Mark Wielaard : afternoon :) [14:29] Jose E. Marchesi : /me caffeinates [14:32] nathan sidwell : oo, caffeine, good plan [14:35] Elena Zannoni : morning [14:35] Elena Zannoni : video from yesterday is up [14:36] Elena Zannoni : i will add the chat now to the website as well [14:37] Ben Woodard : Yeah Caffeine [14:47] Elena Zannoni : hi nick! long time no see [14:48] Simon Marchi : Yes! [14:49] YouTube Live : This meeting is streaming live on YouTube [14:50] Nick Alcock : Elena: yeah it's been a dreadful long five minute absence [14:50] Mark Wielaard : So we are live streaming a hat... OK :) [14:51] Jeremy Bennett : Good morning all [14:51] Nick Alcock : this needs a new video codec specialized for red-coloured hat streams [14:51] Jose E. Marchesi : I hope Nick will remember the hat is there before he sits on his chair again [14:52] Nick Alcock : /me discovers that org-mode has fubared his speaker's notes in places. oh well, will fix later (before the talk) [14:54] Mark Wielaard : Ah that looks odd. I moved the window to the other monitor so I can more easily have it side by side. But now it looks like I am watching something completely different when I put my camera on, because I look sideways. [14:55] Nick Alcock : Yeah, I'm not sure how to consult speaker's notes without looking weird. I suspect it's impossible [14:57] Mark Wielaard : They are talking about https://libre.computer/products/boards/aml-s805x-ac/ [14:57] Elena Zannoni : Nick Alcock: I was referring to the other Nick from the UK :-) [14:57] Sedat Dilek : good afternoon [14:58] Nick Alcock : Elena: too many, it's so confusing. what happened to names being unique property of only one person ever [14:59] Mark Wielaard : My hardware overheated twice yesterday... sniff Binutils BoF ------------ [15:00] Mark Wielaard : I love BBB but building gcc with -j16 is nicer on the old machine than having this video thingy :) [15:01] Peter Zijlstra : the kernel expicitly blacklisted gold [15:02] Dimitri Ledkov : link time & link RAM usage & LTO are a problem. [15:02] Dimitri Ledkov : I have been disabling LTO to get things to build. [15:03] Kees Cook : gold had too many recurring bugs [15:03] Kees Cook : (for the kernel) [15:03] Dimitri Ledkov : Nick you must use headphones due to audio feedback. [15:04] Dimitri Ledkov : you look awesome with pilot like headset! [15:05] Dimitri Ledkov : I'd love for `ld` to do more things that gold did. [15:06] Jim Wilson : https://lore.kernel.org/lkml/CAHbf0-GyQzWcRg_BP2B5pVzEJoxSE_hX5xFypS--7Q5LSHxzWw@mail.gmail.com/T/ [15:06] Dan Aloni : I think any linker maintained on the long-term should have a very good support for compilers' LTO feature, as LTO becomes increasingly common way to obtain maximum performance. Unfortunately this complicates the 'protocol' between linkers and compilers that blurs the distinction between the two on some levels. [15:07] Mark Wielaard : https://gnu.wildebeest.org/blog/mjw/2007/08/31/ian-lance-taylors-linker-notes/ [15:08] Nick Alcock : FYI I *do* hope to look at adding ctf dedup/ctf linking to gold one of these days. Not done yet due to lower-hanging fruit involved in getting ctf generally useful to users. [15:09] Dimitri Ledkov : does it matter? does anyone read them? [15:09] Sedat Dilek : /me reads changelogs [15:10] Andrea Corallo : C-x 4 a :) [15:10] Alex Coplan : I like GCC's approach: write the ChangeLog once in the commit message and then auto-generate the files from those [15:10] Peter Jones : They are rebase conflict farms ;) [15:10] Dimitri Ledkov : i only read them, after my builds fail all over the place [15:10] Tobias%20Burnus : GCC also has ChangeLogs - just they have to be in the git log - and the ChangeLog files are generated by the nightly scripts [15:10] Nick Alcock : Alex: yeah I wrote a script for that too -- and there is also one that goes the other way, from ChangeLog files to autogenerating the commit message. [15:10] Carlos O'Donell : And glibc does this too... [15:11] Segher Boessenkool : what GCC does is the worst of both worlds: not useful, and much more work than without it [15:11] Joseph Myers : I prefer the glibc approach (don't try to write anything in ChangeLog format in commit messages, so less information is included in the generated ChangeLog). [15:11] Tobias%20Burnus : I found it much more convenient than before - also for backporting (cherry picking) [15:12] Nick Alcock : I've had to write too many 200-line valueless refactoring changelogs :( [15:12] Mark Wielaard : I love writing ChangeLog Entries, but hate the ChangeLog files, which is why I like how GCC does it. [15:12] Dimitri Ledkov : I like python style changelogs. They are snippets in individual files. (hand written). But then they never conflict on cherrypicks / merges. Because the final textual changelog is compiled from snippets. [15:12] Jason Merrill : The new GCC system seems to me to produce roughly the same result as before, but is significantly easier to deal with than before. [15:12] Dimitri Ledkov : (cpython specificially) [15:13] Pedro Alves : "When you change the calling sequence of a function in a simple fashion, and you change all the callers of the function to use the new calling sequence, there is no need to make individual entries for all the callers that you changed. Just write in the entry for the function being called, “All callers changed”—like this:" [15:13] Pedro Alves : https://www.gnu.org/prep/standards/html_node/Simple-Changes.html [15:13] Peter Jones : also, as an artifact rather than as a process, 90% of what it gives you is "git show --stat" [15:13] Gary Benson : The thing about changing loads of functions... it's possible to say "All callers updated" or whatever [15:13] Nick Alcock : Dimitri: I worked on a project like that once. After enough time half the codebase becomes nothing but changelog fragments... [15:13] Gary Benson : ah, what Pedro said ;) [15:13] Tom de Vries : /me likes the gcc mechanism, votes to have the same in gdb [15:13] Simon Marchi : I would, but my mic doesn't work today [15:14] Nick Alcock : Gary: ooh that's useful! I guess it applies to data structure revisions too... [15:14] Nick Alcock : ("all users changed.") [15:14] David Malcolm : I spent 3 days writing a ChangeLog for my most recent big rewrite to GCC's -fanalyzer - but in doing so I did find and fix a one-liner bug that was slowing things down 50% [15:15] Simon Marchi : Any form of automation is good, provided it gives reasonable results. I tried running the glibc script to automatically generate ChangeLog entries, and it produced very bad / unuseful results for GDB. Probably because we're using C++ and the script is confused by some C++ constructs. [15:15] Mark Wielaard : Yes, that is my experience too. Writing the ChangeLog entry is a self-review (did I really want to write this this way?) [15:15] Segher Boessenkool : GCC's changelog scripts make trivial changelings a very little simpler to write, and harder ones IMPOSSIBLE [15:15] Peter Jones : @Mark Wielaard - but that's also true of writing a good git commit message [15:16] Joseph Myers : I suspect a large proportion of ld tests depend on details of the output of linking, where there are multiple valid outputs and so the same test doesn't work for different linkers. [15:16] Segher Boessenkool : Peter: if people *do* that, yes [15:16] Mark Wielaard : @Peter Jones right, which is why the ChangeLog Entry needs to go together with the commit message (to explain the what and the why of the commit) [15:16] Peter Jones : @Segher people do that if you don't take their commits when they don't. [15:16] Nick Alcock : Joseph: certainly given the number of things that do basically diffs of objdump output etc (with no real knowledge of the format of what it's diffing) [15:16] Segher Boessenkool : people do *not* yet write good commit messages [15:16] Peter Jones : In other communities people do. [15:16] Sedat Dilek : http://who-t.blogspot.com/2009/12/on-commit-messages.html [15:17] Nick Alcock : Basically my problem with changelogs is that they explicitly deprecate rationales, when that is *precisely* what we want in a good commit message. [15:17] Joseph Myers : In glibc the commit message is something to review alongside the patch. [15:17] Peter Jones : @Joseph: right, and same in kernel and some other projects [15:17] Nick Alcock : They encourage "what", but the VCS can give you that: what matters is "why" [15:18] Jason Merrill : now in GCC the commit message should have rationale followed by changelog entries [15:18] Mark Wielaard : @Nick Alcock the vcs gives you the what that was committed, the ChangeLog entry gives you the what that was intended. [15:19] Segher Boessenkool : marl: exactly, that is why it is so very useful for maintainers [15:19] Segher Boessenkool : marc, even [15:19] Nick Alcock : Mark: ... so this is protecting against, what? VCS bugs? At enormous continuing cost? [15:19] Segher Boessenkool : k [15:19] Mark Wielaard : @Nick no protecting about people bugs :) [15:19] Dimitri Ledkov : I have desires for better support for COFF/PE targets for ARM & RISCV, for UEFI binaries parsing. [15:20] Mark Wielaard : aka, look at what you are about to commit, write the ChangeLog Entry to explain what you really intended. [15:20] Nick Alcock : that's why we have a testsuite. I'd think. Why checks the changelogs against git log --stat? [15:20] Peter Jones : @dimitri feel like helping me with that? I have a branch [15:20] Mark Wielaard : self-review basically [15:20] Joseph Myers : I think it's good that cgen is now in its own repository, since it doesn't depend on any of the binutils code. [15:20] Dimitri Ledkov : @peter jones => yes! i want to help, but wasn't sure how / where. [15:20] Nick Alcock : Mark: "what you intended" is fine -- that's a rationale. Duplicating git log --stat... not so much. [15:20] Marek Polacek : Mark, you can still do that even if ChangeLogs aren't mandatory [15:20] Dimitri Ledkov : @peter jones => i have to hand assemble things at the moment. [15:21] Peter Jones : https://github.com/vathpela/binutils/tree/efi-app-aa64 <-- kinda sorta working sometime before I became someone who works full time on BootHole CVEs [15:21] Peter Jones : (definitely a work in progress) [15:21] Alex Coplan : Any thoughts on the idea of "GAS as a shared library" (e.g. to be consumed by libgccjit, or to facilitate assembling GCC's output in-process)? [15:21] Nick Alcock : (and yes self-review is really good! I do it routinely before committing anyway.) [15:21] Mark Wielaard : @Nick or to review what someone else wrote. OK, your ChangeLog Entry says you changed x to y because z, but you never mention the change in a, is that correct. [15:21] Peter Jones : (Possibly I need to push some local updates to that tree) [15:21] David Malcolm : @Alex Coplan: I have some old bit-rotten patches for that somewhere [15:21] Nick Alcock : Mark: true enuff. [15:22] Mark Wielaard : But I guess I basically like them because I don't trust myself :) [15:22] Carlos O'Donell : It is a bundled gnulib. [15:22] Nick Alcock : II suspect they are a fairly high barrier to new contributors, is all. [15:23] Segher Boessenkool : good commit messages are *way* better than changelogs. good changelings are *way* better than bad commit messages. [15:24] Peter Jones : @Dimitri oh I had one more commit titled "These are an entirely wrong set of patches to add riscv64". Note that these are exactly the sort of commit messages that I would *not* leave in patches sent upstream ;) [15:24] Peter Jones : @Dimitri (pushed now) [15:24] Mark Wielaard : I would love a session on good integration of gnulib [15:24] Sarah Cook : If you want to carry on this conversation, I am in Hackroom 2 [15:24] Jose E. Marchesi : @Mark we can ask Bruno for advice [15:25] Nick Alcock : Yeah: if I ever multithread libctf linking I'll be needing that too, so seconded. [15:25] Tom de Vries : there's already a thread on gdb@ / binutils@ titled "Auto generate ChangeLog for binutils commits" , see https://sourceware.org/pipermail/gdb/2020-May/048597.html [15:25] Mark Wielaard : Some projects use it like gdb by vendoring in the code, but others need a bootstrap that magically pulls it in. [15:25] Peter Jones : @Dimitri anyway this is a path to adding the very basic stuff to get us the same functionality as efi-app-x86_64 and similar, which is still pretty bare bones [15:26] Ben Woodard : oh well [15:27] Simon Marchi : Instead of keeping a gnulib import in the tree, we could make GDB's build process get and create the import dynamically (using git submodules or whatever) [15:27] Simon Marchi : I don't think it would change much [15:27] Jose E. Marchesi : @Simon thats what we do in poke [15:27] Pedro Alves : submodules are the devil. [15:27] Jose E. Marchesi : our bootstrap calls gnulib-tool several times [15:28] Mark Wielaard : Right, but that makes things really much more difficult for occassional contributors [15:28] Nick Alcock : agreed: gnulib-tool is damn slow but it's a thousand times better than submodules! [15:28] Tim Ruehsen : What we do in Wget/Wget2 as well [15:28] Segher Boessenkool : florian: yes [15:28] Simon Marchi : I like submodules :) why are they evil? [15:28] Jose E. Marchesi : Yeah, they work well for things like this [15:28] Peter Jones : in our grub builds we clone and checkout a specific tag of gnulib in our bootstrap script [15:28] Peter Jones : (or do it with a tarball in the rpms) [15:28] Simon Marchi : it's not gnulib-tool vs submodules, it would be gnulib-tool plus a submodule [15:28] Nick Alcock : they disrupt rebases, cherry-picks, merges, and are painful to update as well. [15:29] Dimitri Ledkov : @simon it is trivial to destroy history with submodules which is impossible to recover [15:29] Simon Marchi : To guarantee that a given sha1 of binutils-gdb uses a specific sha1 of gnulib [15:29] Jose E. Marchesi : all these problems are irrelevant to this [15:29] Josh Stone : rust uses submodules, and it's a real pain for coordinated updates [15:29] Jose E. Marchesi : the gnulib submodule is never altered/modified/updated GLibC BoF --------- [15:30] Mark Wielaard : It is a bit like whether or not to checkin generated auto* files [15:30] Nick Alcock : it messes up everyone who does pulls (they all have to remember to update ths submodule too, and no-oneever does). [15:30] Simon Marchi : Anyway, running gnulib-tool and checking in the result is fine. It also helps in seeing the effective diff when you upgrade gnulib, or import a new module [15:30] Jose E. Marchesi : @Nick the bootstrap script takes care of that [15:30] Mark Wielaard : although in that case it is simpler because almost all systems have the autotools installed already [15:30] Jose E. Marchesi : people don't ever have to git init sumbodules manually [15:30] Simon Marchi : `git config --global submodule.recurse true` [15:30] Pedro Alves : it's telling that llvm switched to a unified tree and not submodules. [15:31] Laura Abbott : and 17 people watching on YouTube! [15:31] Simon Marchi : @Pedro: I agree it's not nice when you are working on both repos [15:31] Mark Wielaard : @Jose I take that back. Finally tries to bootstrap poke on this machine... ./bootstrap: Error: 'automake' version == 1.13.4 is too old ./bootstrap: 'automake' version >= 1.16 is required [15:31] Simon Marchi : But when importing someone else's repo that you seldom contribute to, it's fine [15:32] Pedro Alves : I've even vote for putting binutils gdb and gcc back in the same uber tree. It would make it soooo much easier to share code between the projects. Like llvm. [15:32] Simon Marchi : Agreed [15:32] Mark Wielaard : (And yes, I have guix installed, so can pull in the latest easily, but still...) [15:33] Jose E. Marchesi : back to src :) [15:33] Nick Alcock : pedro: either that, or make libiberty git-subtree-managed or *something*. Anything [15:34] Florian Weimer : And integrate glibc as well (the unwinder …). [15:36] Laura Abbott : I find synchronous patch review to be painful except in specific instances. If there are things that need a lot of back and forth it's painful. [15:37] Laura Abbott : What's being described is more patch review triage not patch review itself though [15:37] DJ Delorie : right, it's more important to *remember* to review the patches than to actually *do* the reviews together [15:38] Dimitri Ledkov : What sort of automation can distros integrate? [15:39] Dimitri Ledkov : for example, Ubuntu contributes automated pull requests reviews on github for systemd [15:39] Mark Wielaard : yeah... sigh From mangling... [15:40] Ben Woodard : incorporating the social aspect of the contributor to build the community is a great idea. [15:40] Mark Wielaard : can we get sourceware/mailman fixed? That would help with automating some of this. [15:40] Dimitri Ledkov : fixing mailman => breaks spf records and makes everything mark it as spam [15:41] Dimitri Ledkov : https://github.com/getpatchwork/git-pw & https://github.com/getpatchwork/pwclient right? [15:41] Carlos O'Donell : Yes. [15:42] Carlos O'Donell : git-pw and pwclient [15:43] Laura Abbott : The CKI kernel testing project has done a lot of work to automatically test kernel patches and it was backed via patchwork. [15:44] Mark Wielaard : @Laura do you happen to have a reference/URL to their setup? [15:45] Laura Abbott : I'm trying to find it [15:49] Dimitri Ledkov : Travis has s390x now [15:49] Jim Wilson : FYI Alistair Francis is doing a rv32 glibc talk in the RISC-V MC tomorrow. [15:49] Dimitri Ledkov : and Ubuntu submits s390x results for systemd [15:52] Simon Marchi : If you want to talk about rseq, Mathieu Desnoyers is in the conference, just not in this room [15:53] Carlos O'Donell : We actually know what we want for rseq. [15:53] Carlos O'Donell : We want to hammer out the backwards compatible expansion discussion. [15:53] Simon Marchi : (probably talking about kernel stuff I don't understand) [15:53] Simon Marchi : If you want to talk to him by voice, maybe ping him on IRC to find a time that works for everybody [15:54] Florian Weimer : Probably should happen on mailing lists. I have some ideas, but no code yet. [15:55] Carlos O'Donell : Thank you everyone for coming! [15:56] Olivier Dion : @Carlos: Is there a chat where we can continue discussion on the other stuffs? For rseq, Mathieu is connecting soon and might want to discuss its integration to glibc [15:57] Elena Zannoni : the notes will be in the linux plumber website as well [15:57] Elena Zannoni : yesterday's are up already [15:57] Ben Woodard : Carlos you needed a longer BOF [15:57] Ben Woodard : Is it going to be continued in one of the hacker rooms [15:57] Olivier Dion : okay sounds good C++ modules BoF --------------- [16:08] Frank Ch. Eigler : "sadness will result" [16:09] Jeremy Nickurak : ("no boom today, boom tomorrow" -- "cap: I understood that reference!" ;)) [16:11] Carlos O'Donell : Just delete gthread.h! [16:11] Carlos O'Donell : ;-) [16:13] nathan sidwell : yay jeremy! [16:14] Welcome to the Linux Plumbers Conference 2020 GNU Tools track/Virtual-Room!

Please remember that the LPC anti-harassment policy applies to interaction in this room, and that this room is being recorded.

Please keep your microphone muted and your video off except when participating in the discussion.

This server is running BigBlueButton. [16:14] Jeremy Bennett : Hi Nathan [16:20] DJ Delorie : why aren't we consistently using the intNN_t types? [16:24] Ben Woodard : The thing that we're seeing is compilation time is not really the problem. It is link time. [16:24] Peter Jones : @DJ probably because of the C vs C++ lag on adopting them? [16:25] Carlos O'Donell : *clap* *clap* [16:25] Ben Woodard : ditto [16:26] DJ Delorie : @Peter: sigh, they look like obvious solutions to the "what size is this" problem [16:26] Carlos O'Donell : Lightning talks. We need more lightning logos :-) [16:27] Carlos O'Donell : Perfect! [16:27] Frank Ch. Eigler : "I think you are perfect." [16:27] Mark Wielaard : More people should say that [16:27] Bill Schmidt : Specifically, they should say it to me! [16:28] Carlos O'Donell : Which bug? [16:28] Arjun Shankar : 26383 [16:28] Arjun Shankar : Sorry! [16:28] Carlos O'Donell : /me looks [16:28] Mark Wielaard : sourceware.org/PR26383 Lightning talk: Fuzzing iconv ----------------------------- [16:30] Carlos O'Donell : Go! 5 min! [16:32] Peter Jones : utf-8 not specifying a canonical way to decode any arbitrary arbitrary bitstream was such a terrible mistake. [16:37] Josh Triplett : I see a bunch of glibc folks in this chat, and I just want to say *thank you* for your work on glibc. [16:40] Dimitri Ledkov : This was a fun talk! thank you. [16:40] Mark Wielaard : except that instead of fuzzing, it seems you got full coverage [16:40] David Malcolm : It's cool to see a case where brute-forcing the test space was feasible [16:40] Mark Wielaard : right! [16:41] Frank Ch. Eigler : you just need a beowulf cluster [16:41] Mark Wielaard : showing your age again Frank! [16:41] Dimitri Ledkov : we can =) [16:41] Arjun Shankar : Mark, it is full coverage but only for 2 bytes of input! [16:41] Jeremy Nickurak : 5-minute break on the schedule first, right? [16:43] Mark Wielaard : @Arjun so question, do these converts have much (any) state that would uncover other code paths if you go beyond 2? [16:44] Mark Wielaard : /me assumes they are basically table lookups, but maybe that is too simplistic [16:44] Arjun Shankar : @Mark, yes, they do have state. I don't know too much about it because I didn't have to get into that bit of the code. [16:44] DJ Delorie : @Mark the test would still be running... Lightning talk: Linking LTO and Make ------------------------------------ [16:45] Arjun Shankar : I believe that some converters have shift characters that change how the conversion happens. [16:45] Arjun Shankar : Maybe someone can correct me if I'm wrong. DJ, Carlos? [16:45] Jose E. Marchesi : The BBB should include a watch in the session screen [16:46] DJ Delorie : @Arjun: yes, some character sets have shift-state. I don't know how much they affect multi-bytes though [16:51] DJ Delorie : what about deadlock when a single internal task requires more jobs than are available? [16:52] Josh Stone : rustc and cargo also deal with jobserver tokens for parallelism [16:52] Nick Alcock : yeah it's more or less a general protocol by now (it helps that it's dead simple!) [16:53] Robert Beckett : I wonder if this will help with whole OS build systems (yocto, buildroot) where make -j clearly doesn't limit as requested when a single submake is building large packages [16:53] Nick Alcock : (There is also an implementation of the protocol for ninja) [16:54] Josh Triplett : It'd be nice if things called by make could reach out to make to get the jobserver, rather than the Makefile having to specify that a specific program should be passed the jobserver. [16:54] Nick Alcock : it's an fd, so a unix-domain socket seems like the right approach [16:54] Josh Triplett : Yeah, exactly. [16:55] Peter Jones : make does support modules, but that code has rotted quite a bit [16:55] Josh Triplett : And then Make could *always* export that in an environment variable, to all jobs. [16:55] Nick Alcock : the question is, as ever, naming: I guess the approach would be to have make instantiate a random name, and pass that name down via an env var [16:56] Nick Alcock : (socket naming, to be clear. probably in the abstract namespace? or even a well-knownname in the build tree :) ) [16:56] Josh Triplett : Definitely not in the build tree, that could cause problems, and for that matter "which build tree". [16:56] Josh Triplett : I think it'd make sense to just make up a random socket and put the path in an environment variable. [16:56] Nick Alcock : the objtree :) we know make can write there. a kind of target maybe?? eww this is feeling ugly,a random name is better [16:56] Frank Ch. Eigler : isn't that what the make -j already does? [16:56] Peter Jones : @Josh yeah that is way better [16:57] nathan sidwell : yes, a random socket and env variable is the way I went in my module poc hack [16:57] Jeremy Nickurak : presenter audio has been surprisngly good through the whole conference so far! [16:57] Nick Alcock : don't worry, I have a wind apocalypse going on outside that may well make my talk impossible to hear [16:57] Josh Triplett : Shouldn't necessarily be abstract (that'd be OS-specific), just put the full path in the variable. It'd be possible to support abstract sockets by putting @/path/foo in the environment variable, and expecting applications to translate the @ to a NUL. [16:57] nathan sidwell : frank, no. Make gies you an env variable and then might make the fds it mentions real for you (if it knows you;re a jobservrer-aware program) [16:58] nathan sidwell : frank, and you still have to do the actual spawning yourself. [16:59] Frank Ch. Eigler : why doesn't it make it real preemptively, without jobserver-awareness designation? ld.so in the 2020s ------------------ [17:00] Josh Triplett : The current jobserver protocol relies on inheriting pipe file descriptors from Make, and passing those to a program that doesn't expect them can break things horribly. [17:00] DJ Delorie : @Frank passing open fds to child jobs is dangerous [17:00] nathan sidwell : as josh says, somewhat brittle [17:00] Josh Triplett : By contrast, passing an environment variable pointing to a UNIX socket isn't dangerous. [17:00] Frank Ch. Eigler : aha [17:02] Josh Triplett : One interesting question: could we use UNIX sockets *directly* as the jobserver pipe, or would there be any advantage to receiving pipe fds via SCM_RIGHTS? [17:03] Nick Alcock : what about e.g. nixos / guix? [17:03] Nick Alcock : massive numbers of subdirs, all symlinked to /usr/lib or /usr/lib{32,64} [17:04] Peter Jones : ~/.local/ld.so.cache surely ? ;) [17:04] Nick Alcock : you do need *a* cache, though it doesn't need to be systemwide [17:05] Peter Jones : @Nick: "you do need a cache" is a bug tbh [17:05] Nick Alcock : but if it's per-user, how does ld.so know when to invalidate it? mtimes on directories maybe? [17:05] Nick Alcock : Peter Jones: on systems like guix, just scanning the libdir can take half a minute. [17:05] Peter Jones : @Ben more side discussion than questions really, don't let us stop you ;) [17:06] Nick Alcock : agreed :) [17:06] Carlos O'Donell : Don't call it a cache :-) [17:06] Tree Davies : A per user cache is an interesting idea [17:06] Carlos O'Donell : Call it an "application library index" [17:06] Nick Alcock : best picture of the conf so far [17:06] Peter Jones : call it "preferred link choices" [17:07] Carlos O'Donell : Sure. [17:07] Nick Alcock : YES. automate ABI update detection via libabigail! [17:07] Carlos O'Donell : It's not a cache because you inject search paths into the cache that don't exist in the normal search paths (Debian/Ubuntu do this). [17:07] Peter Jones : It is more or less impossible to get symbol versioning correct in a library. [17:08] Nick Alcock : oh yes they do. abi upgrades are routinely piecemeal in debian etc [17:08] Mark Wielaard : Peter Jones why is that? At least in a basic C library it seems not too bad to use symbol versioning [17:08] Nick Alcock : (er I mean soversion bumps) [17:09] Dimitri Ledkov : there is no symbol version for the struct a symbol accepts...... [17:09] Mark Wielaard : I can imagine it is much more difficult in other languages though [17:09] Nick Alcock : Dimitri: oh yes. shared data structures in general, with the malloc arena being the ultimate version of that. [17:09] Mark Wielaard : Dimitry ah, yes, the version attaches to just the symbol [17:10] Carlos O'Donell : Versioning data is even harder. [17:10] Jeff Law : For C/C++ using the new gcc symbol versioning attributes seems like the way to go, rather than the asm sillyness most things currently do [17:10] Dimitri Ledkov : and when we had abi breaks due to compilers, we have rename packages from libfooA to libfooAgcc5 despite the filename remaining the same (libffo.so.A) [17:10] Nick Alcock : I fear doing this *properly* is an exponential problem, though, or worse [17:10] Peter Jones : @Mark 1) link maps are weird, 2) you can't effectively maintain the symbol versions at the same place as their definitions, 3) you wind up having to maintain the aggregate soname version separately from maintaining symbol versions, and then the @@ @ etc syntax are hard to understand and poorly documented, 4) interactions with -no-undefined are painful [17:10] Dimitri Ledkov : to do "library transition" [17:10] Mark Wielaard : To work around that you basically should only export opaque handles, not concrete data structures, but yeah, then you have to add getter/setters for everything... [17:11] Peter Jones : @Mark and really: because our toolchains give you a pile of unrelated tools to build it yourself, when mostly they should do it /for/ you. [17:12] Carlos O'Donell : @Peter Jones, I don't disagree we should have a way way better integrated symbol versioning infrastructure. [17:12] Mark Wielaard : yes. gcc now finally has a symver function attribute, but it could be improved [17:12] Peter Jones : @Mark look at the change history of src/lib*map.in and src/lib*.abixml in https://github.com/rhboot/efivar/tree/master/src if you'd like to see a good real-world example of me trying it. [17:13] Nick Alcock : yeah: I'm wondering if symbols are even the right level, though -- the data structure problem means that symbols possibly need *multiple* versions, inherited from the versions of the *structures* they use [17:13] Nick Alcock : untrodden snow... [17:13] Jeff Law : versioning structures is a problem unto itself... [17:14] Joseph Myers : Cf. past discussions of possibly generating the installed glibc headers, and the namespace issues that might be helped by making the headers embed the required symbol versions for glibc symbols at application compile time not link time. [17:14] Peter Jones : @Nick the number there is really just a generational marker for a subset of that graph [17:14] Mark Wielaard : drat you cannot click on the links in the presentation [17:14] Nick Alcock : yeah. good point. i.e. the tools should be generating this stuff for you from some higher-level description, perhaps [17:15] Carlos O'Donell : @Nick Alcock, Yeah. [17:15] Peter Jones : Or make the compiler able to keep track of it like it does make dependency generation, so you can keep a result file like an abixml in your source tree, and make -fsyntax-only able to tell you when you broke the abi [17:15] Frank Ch. Eigler : @Mark, check the uploaded copy at the conference schedule site. [17:15] Tobias%20Burnus : Links - try: https://linuxplumbersconf.org/event/7/contributions/756/attachments/536/1059/LPC_Loader.pdf [17:16] Dimitri Ledkov : i hate the dispatcher for ifuncs, we all want the hwcaps dispatcher..... [17:16] Mark Wielaard : nice! thanks [17:16] Patrick McLean : Some gcc optimization options change calling conventions in a way that can break ABI [17:17] Peter Jones : @Patrick yes, and if you change those and it breaks the ABI, you've broken the ABI, so -fsyntax-only should, again, be able to tell you you've done that [17:17] Jeff Law : sure -- what do you want to see happen patrick? [17:17] Dimitri Ledkov : also sometimes one wants input length based dispatcher. Ie. if input is more than this long, yeah use avx512, otherwise no point. [17:17] Peter Jones : (okay so the .abixml-like thing needs to get -grecord-gcc-flags results in it) [17:17] Nick Alcock : I actually wrote something to debug alternate dlmopen namespaces. It *really* does dig into the guts of glibc in horrific ways though [17:18] Peter Jones : don't get me started on dl info link maps [17:18] Carlos O'Donell : @Dimitry Ledkov, You need to switch to more dynamic dispatch in that case e.g. __builtin_cpu_is() [17:18] Carlos O'Donell : @Dimitry Ledkov, You can do such dispatch with sufficient profiling and get it performant. [17:18] Nick Alcock : read and weep (not GPL but you do *not* want to do this so that's not a problem!!!) [17:19] Patrick McLean : I am not sure what the solution, maybe it would be nice to have a way to map the ABI calling convention in to the symbol names.Having a glibc optimized for znver1 can cause some binaries to just crash, which is not great. [17:19] Dimitri Ledkov : @carlos those builtins are usually x86 only, i need arm/power/s390x too! [17:19] Carlos O'Donell : @Nick Alcock, Well... you are missing rtld_db from Solaris in Linux :-) [17:19] Nick Alcock : that's what this was reimplementing (part of, not very well) [17:20] Carlos O'Donell : @Dimitri Ledkov, They don't exist because we haven't pushed on them. I think we could get them implemented as people requested the RFEs for them. I agree though, I did notice recently that access to certain builtins was x86 only. [17:20] Frank Ch. Eigler : @patrick .... auto-tagging symbol names with a hash of the transitive abi of its declaration types [17:20] Carlos O'Donell : @Nick Alcock, I figured that was the case. I think we need an rtld_db in Linux, but the in-process stub library has issues we know about and would like to avoid with a more modern design. [17:21] Dimitri Ledkov : @carlos i thought there was opposition to cross-arch-supporting builtins. I guess i should propose some of them then. [17:21] Josh Stone : @Frank see Rust symbols for that kind of hashing... (lack of ABI aside) [17:21] Peter Jones : @Carlos if you're collecting a wart list here, every link map entry should have data about what entries in /proc/self/maps it corresponds to [17:22] Nick Alcock : Can't you figure that out from the corresponding addresses? [17:22] Peter Jones : you very much can not [17:22] Carlos O'Donell : @Dimitri Ledkov, At the lowest level you need these builtins. Ask for them if they enable your workload. We are shipping per-arch APIs and ABIs e.g. sys/platform/ppc.h [17:22] Patrick McLean : @frank one could theoretically autogenerate wrappers for other ABIs, that changes the calling conventions to the "native" ABI of the library [17:23] Peter Jones : you can't distinguish dlopen() results from mmap() results, you can't tell which things are .bss from two invocations of dlmopen() with different link map ids [17:23] Peter Jones : etc [17:23] Frank Ch. Eigler : a crazy idea indeed [17:23] Ian Forbes : more audit hooks [17:24] Nick Alcock : oh true! I was sigging around in private guts for that, but of course we are trying to get *away* from that [17:24] Nick Alcock : s/sigging/digging/ [17:24] Dimitri Ledkov : the one thing i want is to be able to build cache for a given LD_LIBRARY_PATH such that if hash of ld_library_path is "foo" and there is ld.so.cache.foo use that [17:25] Peter Jones : an API to tell me what the /effective/ LD_LIBRARY_PATH is (and possibly modify it on a per-linkspace handle basis) would also be very useful. [17:25] Peter Jones : that'd really help build better tools for e.g. initramfs generation [17:25] Ian Forbes : namepace cloning/shring [17:25] Peter Jones : which is really just container generation these days [17:25] Dimitri Ledkov : oooh buildid. cool. [17:25] Frank Ch. Eigler : thanks ben! [17:25] Mark Wielaard : Thanks Ben! [17:25] Carlos O'Donell : *clap* *clap* [17:25] Frantisek Sumsal : Thank you! [17:25] Peter Jones : thanks Ben! you've really got us getting lively discussing this [17:25] Peter Jones : *clapping* [17:26] Ben Woodard : @peter jones yoou are welcome. I would love to take this somewhere else if you are interested. [17:26] Mark Wielaard : Nice sound effects [17:27] Peter Jones : I am and I think others are as well, but I also want to see @Nick's stuff because CTF is *SYMBOL FOR HEART EMOJI* [17:27] Dimitri Ledkov : can you turn video off [17:27] Dimitri Ledkov : to increase bandwidth for the audio [17:28] Aaron Sawdey : YEs this sounds like a bandwidth problem. [17:28] Peter Jones : I think Nick is saying the wind is causing the audio codec for his microphone to have a bandwidth problem [17:28] Dimitri Ledkov : EMC interference =) [17:28] Aaron Sawdey : this is better now [17:28] Pedro Alves : maybe use the laptop mic / avoid bluetooth? [17:28] Peter Jones : yeah that does seem better [17:28] Mark Wielaard : New frontiers in CTF linking ---------------------------- [17:31] Daniel Xu : Is there a desire/intention to support C++ and other languages? [17:33] Elena Zannoni : @Daniel Xu... wait until last slide :-) [17:39] Frank Ch. Eigler : @Ben Woodard, hashing TUs ... sounds a little like an ABI checksum. [17:40] Ben Woodard : We are going to need a checksum. [17:41] Ben Woodard : I'm going to have to look carefully at this. Most of our code is C++ and Fortran though. However at the exported interface level where the laoder works, that may not matter. [17:41] Ben Woodard : What I was thinking was to add something like: [17:42] Ben Woodard : DT_INTERFACE immediately after DT_NEEDED in the ELF spec. [17:42] Dimitri Ledkov : what does ocaml / ghc (haskell) do? they seem to have some kind of an abi checksums, no? [17:42] Ben Woodard : DT_INTERFACE would use a val which was the offset for the DIE [17:43] Ben Woodard : However in the cache I don't know how to reduce the resulting tree to a hash. [17:47] Peter Jones : Sounds like a merkle tree... [17:48] Dimitri Ledkov : blake3 hash =) [17:57] Mark Wielaard : Thanks Nick [17:57] Nick Alcock : I have no idea how comprehebsible that was [17:57] Nick Alcock : The speaker's notes are much clearer, probably (see the indico entry for the talk) [17:57] Nick Alcock : I thought I was rushing, but no... [17:58] Nick Alcock : @Ben Woodward: CTF could defnitely help here: shared libs could carry .ctf sections, and users could query them to get descriptions of the types they needed, even generate headers from them etc :) [17:59] Ben Woodard : yeah a merkle tree or blake3 hash sounds like a cool idea. [17:59] Nick Alcock : This does make type definitions in headers semi-obsolete :) [17:59] Nick Alcock : @Ben Woodard, this is literally a merkle tree (using sha-1 only because that is what is already in libiberty: we can change that at any time) GCC's -fanalyzer option ----------------------- [18:00] Nick Alcock : the hashes are never emitted, only the dedupped types are [18:01] Nick Alcock : -Wanalyzer* has been *so useful* in libctf dedup development :) [18:02] DJ Delorie : my only concern is that we typically build with -Wall -Werror... false positives are build-stoppers :-( [18:03] Segher Boessenkool : yes, -Werror is a show stopper [18:03] Nick Alcock : Maybe -Werror shouldn't stop the build because of "estimated warnings" like this, and, famously, -Wuninitialized [18:04] Nick Alcock : Always-accurate warnings are obviously different from educated guesses, and mayeb -Werror should take this into account [18:05] DJ Delorie : Except we *do* want to be sure we've analyzed everything we get warnings about. It's a double-edged sword. [18:05] Anatol Belski : Is it possible to configure custom alloc/dealloc routines? Is it possible to mark some routines as noreturn? [18:05] Jeff Law : in my experience its the maybe-thingsies that find the most interesting and hardest for a human to identify bugs [18:05] Segher Boessenkool : yes [18:05] Segher Boessenkool : but they tend to have too many false positives for -Wall [18:06] DJ Delorie : or we need the inline warning controls to be better documented? ;-) [18:06] Jeff Law : having spent a couple years in the distro-rebuilding game now dealing with those and I don't think the false positive rates are a major problem [18:06] Segher Boessenkool : Jeff: they are for people who foolishly use -Werror [18:06] Jose E. Marchesi : Damnright! [18:07] Jeff Law : a good deal of the Fedora and RHEL packages use -Werror [18:07] Peter Jones : @Segher those of us that do that usually are pretty good at -Wno-... for stuff we disagree with [18:07] DJ Delorie : @Jeff but the rate of new warnings is growing. I *like* warnings, but I'm concerned about the impact of the false positives. [18:07] Jeff Law : and every release stuff gets flagged, but the vast majority have been useful to dig into [18:08] Nick Desaulniers : -Wno-error=foo to make -Wfoo not an error when -Werror is set. [18:08] Jeff Law : i understand your concern dj, but my experience is it hasn't been a major problem in the distro builds [18:08] DJ Delorie : @Nick you can do that inside the sources, for the specific line that you've reviewed, too [18:08] Nick Desaulniers : pragmas? [18:08] DJ Delorie : pragmas and attributes [18:09] DJ Delorie : but those are harder to "undo" when you want to check for those warnings, unlike a makefile flag [18:09] Segher Boessenkool : it is 100% impossible to build a non-trivial program that uses -Werror with a. newer compiler than it was tested with [18:09] Nick Desaulniers : agree [18:09] Jeff Law : segher, I do that for thousands of packages... [18:10] Jeff Law : I've already done fedora builds with gcc-11 :-) [18:10] DJ Delorie : @segher agreed but you still want to review what they warnings are about, and -Werror makes sure you do [18:10] Segher Boessenkool : dj: no, only the user himself can make sure he does [18:10] Nick Alcock : oh look my bluetooth just dropped and my wifi bandwidth is about 1/10th normal. either the weather orsomeone with a *really* noisy microwave [18:11] DJ Delorie : right, I mean, how often does the user read the build logs looking for warnings? A hard fail is a self-imposed "force" to check warnings [18:11] Jeff Law : dj, precisely... [18:11] Segher Boessenkool : so this is a "we do not trust the user" flag [18:11] Patrick McGehearty : The first round of cleaning up legacy code can be huge. glibc went through that a few years ago when -Werror was added to the build process. After that, the effort of keeping up is much less. [18:11] DJ Delorie : more like an "I don't trust myself" flag :-) [18:11] Segher Boessenkool : heh [18:12] Josh Stone : "keep me honest" flag [18:12] Nick Alcock : combined cfg/state graphs were big a decade back, but in my experience whenever you do much wioth them you need to break them apart again... [18:12] Jeff Law : or the "hey, maybe the new compiler can find things the old one didn't" [18:12] DJ Delorie : jeff YES [18:13] Segher Boessenkool : well, I read build logs (summaries, anyway) [18:13] Jeff Law : for gcc-11 vs f34 the number of uninitialized memory objects found were significant [18:13] DJ Delorie : so that's what that red dot is... [18:13] Jeff Law : good luck reading logs from 9000+ package builds [18:13] Ben Woodard : The binary rewriting people really would like CFGs the perf tools people are BEGGING for DFG. [18:13] Segher Boessenkool : Jeff: the summaries just say "N warnings" [18:13] Peter Jones : I really wish we had a way to say "warnings are errors by default, but warnings newer than gcc 10.2.1 are warnings by default" [18:14] Jeff Law : and that makes them totally uninteresting [18:14] Jeff Law : pjones. yea, that would be useful [18:14] Marek Polacek : wouldn't be too hard to implement [18:15] David Edelsohn : Jeff: Does F34 patch locally and push the fixes upstream? [18:15] Segher Boessenkool : Jeff: not uninteresting... it is very in your face, so you see if things changed [18:15] DJ Delorie : segher assuming you remember the old number [18:15] Jeff Law : david, yes. I commit the fix locally, and let the package maintainer know they need to engage with upstream [18:15] Segher Boessenkool : diff [18:15] Jamie Lokier : I've been thinking of building a 3D interactive tool (basically a mini game engine) for visualising & interacting with these enormous multi-scale graphs. Feels, in my imagination, like it might help. [18:16] Ben Woodard : Go all the way - make it a VR Experience [18:16] Jamie Lokier : Ben: Exactly. [18:16] Nick Alcock : Jamie: I'd use it [18:16] DJ Delorie : the biggiest dependency graph I've ever worked with, even printed five feet tall, was a large fuzzy grey dot. [18:16] Nick Alcock : with collapse/expand on each node [18:17] Jamie Lokier : VR is cheesy but for this I think it might aid understanding. If done right. And cause brain freak-outs if done scarily wrong :-) [18:17] Stephen Crane : ooh that's a cool idea [18:18] Stephen Crane : but getting that to not be overwhelming sounds tricky [18:18] Jamie Lokier : DJ: Yeah, at some scale it starts to become a texture or fluid. Need tools to make sense of it then. [18:19] Nick Alcock : deduplication of nightmare graphs everywhere! [18:20] Ben Woodard : We need a library for that. [18:20] Ben Woodard : Graph manipulation and deduplicationo. [18:21] Nick Alcock : everyone has wildly different graph representations, is the problem... heck ctf's is almost *implied* (we hop from hash values to input ctf_file_t's and back for every node) [18:24] Ben Woodard : Shockingly Boost doesn't have a dedup [18:24] Ben Woodard : https://www.boost.org/doc/libs/1_66_0/libs/graph/doc/index.html [18:25] Nick Alcock : of course, generalized dedup has horrible time complexity [18:25] Ben Woodard : The future is parallel [18:26] Nick Alcock : the only reason libctf's is tractable is that we exploit knowledge of the C type system :) [18:26] Nick Alcock : most cycle-detection algorithms, let alone dedup are... not very parallelizable :( [18:26] Elena Zannoni : thre is no next session [18:26] Jason Sidebottom : Sounds great David, looking forward to testing this out. [18:26] Elena Zannoni : you can continue [18:26] Nick Desaulniers : David: I'd love to follow up/ask questions, but I'm having issues connecting with my mic in this room (there's a thread in #helpdesk, others can repro in this room). Any chance we could follow up in a hackroom? https://meet.2020.linuxplumbersconf.org/hackrooms / virtual beers? [18:27] Elena Zannoni : @david you can continue here [18:27] Jamie Lokier : Graph isomorphism is hard in the general case, so general-purpose graph dedup is tricky. Though possibly quite useful. [18:27] Nick Desaulniers : cant join audio here [18:27] Jamie Lokier : Great talk Dave! [18:28] Tobias Burnus : Same problem here - listing mode works, but joining with micro support didn't not work a short while ago :-( [18:28] Nick Alcock : Jamie: yep. maybe what we need is a general thing one can add constraints to to speed it up :) pie-in-the-sky stuff though [18:28] Elena Zannoni : @nic Desaulniers: type here [18:28] Nick Desaulniers : audio is working for me in hack room 2, can't join with mic here [18:28] Nick Desaulniers : plz no [18:28] Elena Zannoni : feel free to swtich room [18:29] Mark Wielaard : Thanks David. Great talk and there were indeed some false positives in elfutils, but it found various real issues. Unfortunately really need to get me some food now or I faint... [18:29] Elena Zannoni : hack rooms are not recorded though [18:29] Elena Zannoni : let me see if i can start the recroding there [18:30] Jamie Lokier : EBPF does support "bounded loops" [18:30] Serhei Makarov : BPF brute-forces loops IIRC. It can't really be heuristic. If it misses a potential error that's a potential security flaw. If it can't verify all code paths it must reject the program [18:33] Nick Alcock : *All* oasis specs are like that. It's where conciseness goes to die [18:34] Jamie Lokier : Clearly, output in Emacs Lisp then :-) [18:35] Nick Alcock : JSON -> Just Sexp Output Notation [18:39] Serhei Makarov : There's clearly a niche for an emoji-based language. It could take inspiration from APL [18:40] Peter Jones : @serhei that is literally haskell AFAICT :) [18:40] Guy Lunardi : Sounding good Nick :-) [18:41] Nick Alcock : Serhei: prior art: https://www.stroustrup.com/whitespace98.pdf [18:46] Peter Jones : Might be worth focusing on analysis of what's common between different results [18:46] Aaron Sawdey : "language is a thing defined by what the compiler accepts" sounds like words from somebody with arrows in their back from using c++ in the 90's ... [18:47] Josh Stone : kgraphviewer is decent too [18:48] DJ Delorie : I use dot->graphviz->ghostview [18:50] Elena Zannoni : don;t close the meeting [18:51] Frank Ch. Eigler : bye ! [18:51] Elena Zannoni : bye all [18:51] Gaius Mulley : thank you all [18:51] Sarah Cook : Thank you everyone! [18:51] Serhei Makarov : *applause* [18:51] Jeremy Bennett : I shall capture the chat now for the record