Swift Package Indexing

It's kind of funny, it made researching the packages for this week's episode of the podcast

a little bit more difficult than usual. So what we're talking about is that you may have noticed

if you visited the package index over the last couple of days that readme files are not rendering

on package pages and we had a little bit of technical debt that hit us squarely

in the front of our faces this week. A little bit of background to this is when we implemented the

readme file rendering inside package index, we originally did it with a Markdown processor,

a GitHub-flavored Markdown processor, and so grabbed the raw readme file, rendered it out to

some HTML, and then did some fixes to fix up image paths and things like that.

And then over time there were edge cases and edge cases and edge cases of things

that didn't quite render correctly that we decided we need to have GitHub's

rendering of their readme files really. And so the way that we approached that,

because it was the only way that we thought we could approach it at the time,

was to grab the rendered HTML page that GitHub would serve for the readme, find

the readme file, and then render that through onto our site. And at the point

that we deployed, or just about to deploy that feature out to the site, one of our

contributors, James Sherlock, mentioned that there was an API for this and this

is something that was news to us that we didn't know about. We did know that there

was a readme API, but what we didn't know is that if you specify a

specific accepts header and tell it that you want a rendered version of the

readme information, then it will give you this rendered HTML page in the

correct way, not by scraping, doing it through the API, which is what we should

have been doing. But this feature was implemented. Changing it to the API

version would have been quite a lot of work and it was one of those things that

we put in as an issue. In fact I looked at the issue the other day and it's

from 2021, so a couple of years ago now. And yeah we stuck it in as an

issue and promptly kind of forgot all about it. Right up until GitHub changed

their readme page design, which is always the problem if you scrape anything then

you're vulnerable to if the original page design changes or something like

that then you you have a problem. Now we had some code in there so it fell back

nicely and displayed a link to the readme instead of the actual readme but I

must admit I desperately miss the readme files because it is a real pain to

have to click through every readme file. Yeah there's no there's no worse reminder.

I mean, it was really, it was really harsh.

I mean, I look at package pages often, but, um, I'm not looking for the readmes

every time, but this, you know, this last couple of days, as I, especially as I was

working on it anyway, to, to fix this, every time I looked at a package, I was

like, Oh, God damn it, what should I be doing?

Should I be preparing for the podcast or fixing this?

So it's kind of interesting that, um, when I, when I first noticed this, I noticed

that almost immediately that GitHub must have deployed this change because I was

researching packages for iOS Dev Weekly. It was on Thursday last week that I

first noticed this problem and it was instantly intensely annoying to me that

I couldn't read the reading files on the index and I talked with Sven about it a

little bit and you kind of said, "Ah, it's okay. Well, it's okay for a few days."

And I thought to myself, "This is... I'm not sure it is okay for a few days. It's

Really annoying.

So I'm glad we shared the pain today.

- We certainly did, yeah.

I think part of the difficulty with the readme's

in particular is that it's one of those things

where it's very easy to assume,

well, this is always going to be a readme markdown file,

right, but readme's actually don't have to be.

They can be in a number of different formats.

So it was really a bit tricky to find out where,

This is actually nice that they have a fully rendered HTML version of it that you can get,

because that funnels all the readme formats through a single output format that we can then

tap into. Because I remember, I vaguely recall looking into this at some point and stopping,

because my assumption that it would be a markdown file that we could easily ingest and embed,

because, you know, getting that and adopting it is easy. We're handling markdown already. But then

when I saw, well, actually, you know, it could be different formats. That's when it all stopped and

became more complicated, as it often does.

Yeah. And of course, if we, you know, we still have a plan. It's not a plan that we're working

on actively, but we still have a plan to support other source code providers with the package index.

and certainly GitLab would be a good contender for the second source control provider to support.

But every little reach into the GitHub API like this ties us more closely, so it makes that job

a little harder. So I'm always conscious of that, but this definitely needs fixing.

Yeah, I mean there's to hoping that GitLab, which do have a quite extensive API,

we're using them in our build system so we have some experience with the API.

And my hope is that everything will be available, but you never know, right?

Yes.

There's going to be little bits and pieces that are going to be different or not there.

And if, when the time comes, it'll be interesting how that'll pan out.

Absolutely.

Right, so what other news do we have? I think worth mentioning is that we've crossed 6,000 packages

a couple of weeks ago, I think just after we recorded the last episode.

I actually managed to catch a screenshot of it when it was exactly at 6,000 as well.

We didn't have to edit the package list down to 6,000 to make that happen.

No, it was pure coincidence that I managed to just glance at it because I knew that it was

imminent and so I was checking it every every day or so and I happened to catch

it when it was exactly 6,000 packages. Well there's always our left pad package

to deal with off by one arrows right that we need. Yeah and that's a that's a

huge milestone I mean 6,000 packages it's it's not an enormous number by some

package managers and I'm looking at NPM when I say that but it's certainly not

not an insignificant number and it's something that,

this is 6,000 kind of manually curated packages.

This is 6,000 packages that people have said,

this is something that I would like people to be aware of.

- Artisanal packages, yes, you get them here.

- Artisanal packages.

- In Swift's defense with Codable,

that nerfed probably around another 20,000 packages

that our JSON passes out of existence, so.

(laughing)

- I think there's a few of those still in there, yeah.

- Those 6K are actually worth a lot.

(laughing)

In fact, I did see one, a new JSON package come through this week actually.

There have been a couple, yes, but they do actually earn their existence by adding a

couple of extra bits.

We talked about one, I think last time we recorded or the time before dealing with those

little bits where Codable is falling short a bit for the moment or just in general.

I do have one other little bit of news

that we might want to talk about.

And that was a post on the Community Showcase forum

on the Swift forums, which just actually,

before we talk about the specific post,

I would like to recommend that people check out

that Community Showcase forum.

It doesn't get a lot of new posts,

maybe two or three a week, something like that.

But I get a huge amount of value out of reading

the posts in that forum.

It's a great place to learn about what's going on

in the community.

And I noticed it was actually the week before last,

but there's a post by Neil Jones titled Swift on RISC-V

or RISC-V, which I have been reading the Wikipedia article

and apparently it is pronounced RISC-V.

Now I didn't know a lot about RISC-V

and all I know now is the contents of the Wikipedia article.

So I'm far from an expert on it.

But it's an open standard instruction set,

a reduced instruction set that is being implemented

into hardware and things like that.

It's been around for quite a while.

And Neil is working on a project

to bring RISC-V support to Swift.

And in his own words, the RISC-V ecosystem

growing at a phenomenal rate and 2023 will see the introduction of many new exciting and powerful

devices. So at first I thought, well, is this kind of some embedded, you know, instructions that might

suit an embedded computing or something like that? But it's the whole range of things. It's from

small boards up to really large workstations and things like that. So to continue our talks about

platform support, we may, it's looking like we may need an enormous list of platforms at some point

if this comes to fruition. Oh dear me.

It's just the space on the matrix. It's also, I mean, we just need that stuff.

You know, we need testing for that. It's interesting. I mean, it's great to see that

Swift is branching out like that. Yeah.

What's the vendor? Is that like vendor neutral? Are there specific vendors, multiple vendors that

implement machines?

It's all I know is that it's an open standard and, um, and I believe most of

the work on it is open source as well.

So I, I don't think I didn't see, um, a vendor named when I was reading about it.

Well, I think Apple should just be sending us any, any hardware platform that uses

Swift for compatibility testing.

That should obviously include

He tries to angle for a vision pro headset.

Well, that was my, I wouldn't say no to that, but

I don't believe that runs RISC-V.

It's clearly beneficial.

Well, but it's a platform we support, so...

It is, and I've seen a couple of readmes come through this week

that said added support for VisionOS and things like that, so...

Oh yeah, and we should probably mention, because at the time we recorded last time,

it wasn't live yet, but we have added that and it's completed the processing.

So we're all caught up.

It's all there in the matrix and you can check there if you're not sure about

your VisionOS support of your package.

Yeah.

So, and it obviously, depending on what the package is, a lot of them just

automatically get VisionOS support if they're not doing anything specific to

UIKit or AppKit or something like that.

But, but yeah, you can check your package live on the site as of now.

Exactly.

And the final bit we might want to mention is that the macro search is live now.

And the way that works is you look for a product, product colon macro will list you

all the packages that ship with macros.

Um, and we'd especially like to hear if you find this useful, if you're using this,

we have a bit of, we've had a bit of metrics to see if that's being used because

it is the way we've done it is, isn't quite straightforward.

It's a bit of a performance cost when we compute our materialized view for search.

Um, so if there's not a lot of pickup of that, we might, we might actually take

that out again at some point if we have other things to add, so it's not a, it's

not a zero cost addition, but it is there.

Uh, give it a try.

Product colon macro to see what macro packages there are.

And I think there's a, there's a fair few, I think it was a dozen or so that,

that ship macros.

- I think we've had metrics in there

for whether people are using any kind of filters.

So not obviously just searches,

but filtering searches by platform

or Swift version or whatever.

And I actually don't know how often any of those are used.

My gut feeling would be not very often.

- Not a lot, no.

- Yeah.

- Yeah, that's why I'm curious.

And they aren't used a lot,

and then the product macro search

even, you know, that's even a smaller slice of that already small chunk. And we pay an outsize

price to compute that. I mean, it's not terrible, but it's significant. So that's why I'm a bit

concerned whether that's a long-lived thing. And it's also the way we had to implement that isn't

ideal because we're actually saying macros are products, which they aren't in the strict sense

of the package manifest because they're actually targets and we're sort of faking it a bit.

it would be easy if they were proper products, but that's really an implementation detail.

What's important is to know if people find this useful or if you can think of ways to

expose this feature better. I mean, we do spell out the search extensions below the search box,

so there's a little text explaining what you can do to search in more detail,

but it doesn't seem to be used.

Maybe that's because it just doesn't come up often

or people don't know.

That would be the interesting bit to know.

Whether that's just a case of discovery

or it being really useful.

- I think I'm fairly happy with the way

that we present the discovery of that.

So obviously it's not there on that first homepage,

but as soon as you search,

it's right below the box there and there,

links so that if you even tap on one of those links it pre-fills the search with

a filter so that you can kind of get started with filtering. I think that

search, I think Google has effectively ruined us and what people do for search

is they type in some words and they hit enter. Yeah, yeah that's true. The other

thing is I tend to, especially in places I visit a lot, I tend to turn blind

to extra text on stuff. So I start overlooking small print very easily. Only the first time

on a page and I'm trying to find my way around is when I look at stuff in more detail. And

then once I sort of know where stuff is on a page, I really only zoom into the bits that

I actually need and become blind to all the stuff around. It becomes ornamental to a degree.

It's fascinating how that works.

- Absolutely right.

In fact, I was talking to somebody

about their app the other day

and they'd sent me a test flight link

and I had used the app and pointed out something

that I got stuck with when I was using their app.

And they pointed out to me that it's written right there

on the screen that I got stuck on.

- Oh dear, yeah.

That's the way it goes, isn't it?

- And that's it.

It's just a fact of software is that we,

it's not a software problem, it's a human problem.

We overlook things that are not directly pointed out to us.

But I do have a solution.

I think there is still support in most browsers

for the marquee tag,

so I think we're gonna use the marquee tag.

- The marquee tag?

What is that?

- Oh, the marquee tag was a tag that scrolled text

across the screen. - Oh, right, in a ticker.

- Yeah, that's right. - Right, yes.

- So look for that coming soon.

back to the 90s, excellent.

We might as well add a construction site

little icon to it.

- Exactly, exactly.

Well, it is always under construction, you are correct.

- It kinda is, isn't it?

Right, does that, that wraps up the news section, doesn't it?

- I think it does, yeah.

Should we do some package recommendations?

- Let's do the packages.

What do we have, do you wanna kick us off?

- Sure thing.

So my first package this week is by Sindre Sohus,

and it's a package called doc progress. It's actually been around for five years now, but I

only came across it this week and it's a kind of niche. I think both of my packages this week are

kind of niche, but it's kind of niche in that it's for macOS applications only and it's for

macOS applications that want to show progress of something when the window might not be visible.

So there are many different ways to show progress with this package.

And it rendered takes advantage of the fact that you can render your icon in a

Mac OS app, completely custom code. You know,

you don't have to have an actual icon file at all.

You can just draw into a context and that becomes the icon.

And so what this package does is it has a kind of the first,

the basic one is has a progress bar across the bottom of the icon that fills up

over whatever progress that you're sending it,

or there's one that outlines the icon with a squircle

showing progress in that outline.

The circular ones, there's badges,

there's badges with progress, there's badges with pie charts,

there's lots of different options for showing something.

And it's the kind of thing that is not going to be useful

when someone's actually looking at your application,

but for really long running processes

where people will actually switch away from your app,

this felt like a great way to keep people updated

on whatever it is that your app is doing.

- Nice, yeah, I'm actually staring at one of these right now

because my application folder has a progress bar

underneath it, I guess some app update died.

It's been like this for days now.

- Well, certainly that one might need a reboot.

That's how we fix things, isn't it?

- Yes, exactly, yeah.

But for ones that are not terminally stuck,

this feels like a nice solution.

- Nice, when you said doc progress,

I first thought, oh nice, this is going to tell you

how far along you are with documenting your project.

And then you said five years,

I was like, well, maybe it's something different.

- That would be docc progress, yeah.

- Right, my first package is one

that people may have heard about

because it's a new package by Apple.

It also showed up on the Swift blog and it's called Swift HTTP Types by Apple.

It's a package that collects foundational HTTP types like HTTP request, HTTP response,

and it's for use in client and server packages.

And that's really nice because if you're using like different ways of interacting

with HTTP, you've probably come across quite a number of different implementation of those.

And the goal here is to unify that a bit and use that package as the underlying package

in Swift Neo and Foundation, for example, to make sure you have only one version of

these around and, you know, sort of can use that one implementation and knowing how that

works because they often have like slightly different ways of setting headers, for example,

and stuff like that.

So that's really nice.

And I guess also, you know, it's going to open up to be used in other server-side packages

to adopt this either directly or automatically via Swift Neo.

It's really nice to see more and more of these foundational packages pop up and as individual

packages, you know, not necessarily tied into foundation itself, but they are being there

on their own.

And of course, this is not just interesting

for server-side packages.

Clients also need to deal with this

when they interface with server APIs, for instance.

I just mentioned setting up headers and that sort of stuff.

So it's quite a package with reach, I'd say.

And it's good to see it, good to have it.

- So I'm ashamed to say that I did see this blog post,

but I actually haven't read it yet.

Is this the kind of package

that is going to be purely for use with server-side Swift

and Swift Neo based projects?

Or is it also something that you would use outside of that?

- No, I think you'd use it outside.

I mean, this is going to be the underpinning types

for foundations URL requests.

So when you construct a URL request,

you'll be using eventually, that's not the case right now,

but it will, that's one of the goals is to have that

as the HTTP currency type for URL requests.

So when you construct a URL request,

which you do when you use URL session or allow more fire,

you'll be using this package and it'll be the same type

that you also use on the backend to drive that.

So that's quite nice.

- And so the package itself is just a representation

of those request types.

- Yes, exactly.

Which will then be adopted in those other packages

that I mentioned, yeah.

- That's great.

Well, that's good to see.

And it's good to see that things that,

'cause obviously there's the server side of Swift,

which is primarily open source

or entirely open source maybe.

And then there's the kind of Apple side of Swift,

which is AppKit and UIKit and SwiftUI,

which is primarily closed source.

And of course there are some exceptions

like the new foundation library is completely open source

and that underpins everything in those platforms too.

but it's nice to see more things become open source

that are gonna be used in those platforms.

- Yeah, absolutely, yeah.

- My next package is a package by Darren Ford.

And it's a package that I have confidence

that I will probably never,

no, I'm not gonna say confidence then probably,

I have confidence that I'll never use this package.

But what I really liked about this package

is how I could see the rabbit hole

that Darren went down when implementing this.

(both laughing)

So it's a package called Color Palette Codable,

and it is a codable implementation of color palette files.

The original one is an ASE file,

which is an Adobe Swatch Exchange file,

which is, if you read the read me,

This is actually the file that Darren needed to decode.

So he obviously had a requirement to read one of these files.

And just to be clear what's in these files,

it's a list of colors, names for those colors,

in the, get formed into a palette of colors.

So it was a group of colors.

So he needed the Adobe Swatch Exchange file

and implemented something and thought,

well, maybe I'll make it a package.

And then that, he said that this then extended to ACO,

which is an Adobe Photoshop color swatch file.

And then it expanded to other types.

And there's a smiley face in the read me

because he's implemented support for,

I think 12 or 13 different color palettes implementations,

including PaintShop Pro files,

which was a real blast from the past.

Do you remember PaintShop Pro on Windows?

- Oh, I don't know.

I used Windows a bit, but I switched to Macs in '91,

so I missed out on a lot there.

- So I was Windows up until the early 2000s,

and PaintShop Pro was one of those applications

that was absolutely essential to have on your Windows PC.

Microsoft RIFF, R-I-F-F, Palette Files,

Sketch Palette, which I've never even heard of,

corral draw, every type of color. And I'm confident that even Darren hasn't used in

anger most of these. So I thought as I was reading this read me file, I saw the rabbit

hole you went down Darren and I hope he came out the other side of it.

Nice. It's one of those, if I understand correctly, part of that was to make colors

codable or... Yeah well color palettes codable so it's the combination so as some of these

palettes there's actually a it's a fantastic readme file for what the package does. There's

a table about three quarters of the way down the readme file of all the different files that are

supported. What are they binary? Are they text? Are they JSON text or whatever? Do they support

Do they support named colors?

Do they support named palettes?

Do they group colors?

Do they have color type support?

Do they support color spaces?

So there's obviously a whole load of information

that you can use in these palettes,

but no, it's more than just a single color.

- Right.

The reason I'm asking is because I saw this pop up

in the last week.

I think someone went to the labs at WWDC and asked,

Well, why isn't SwiftUI color codable so they can save color choices to the,

what you call it, the defaults, user defaults.

And the answer is quite interesting because, you know, naively you think,

well, how hard can it be, right?

It's just three numbers, right?

Red, blue, green.

But the answer was, well, actually, you know, it, what actually displays

depends on your display, right?

calibration, that sort of stuff. So it isn't dark and light mode.

So a color red is slightly different depending on whether you have it in dark

and light mode. I believe there's a variant that SwiftUI gives you

depending on context. So a color is context dependent. So when you save

it, how much of that context do you pull in? You know, it's not just that. It's

one of these great examples of a thing that seems to be so simple at first glance, where you'd say,

"Well, why isn't this just codable?" And then, "Oh dear me. Yeah, well."

- If you want to learn more about this, Mark Edwards has a two-part blog post on color spaces,

which is well worth the five to ten minutes it would take to read it.

Nice. We'll add that to the show notes as well. I find stuff like that so interesting.

And I think that's one of those places where sort of experience shows a bit because you become

really a bit more resistant to this. Well, just do this because you've been there a number of times

where you thought, well, of course, this is going to be easy, right? And then like days later,

you think, oh God, what did I get myself into? And I love this sort of stuff where

there's a very, you think, obvious first stab at a thing, and then it opens up this whole world of

complexity that is so neatly hidden. And we sort of get tricked into this a bit as well with our

systems that we like to use, right?

I mean, it's one of these arguments, Linux or Unix versus the Mac, right?

The Mac sort of tends to present itself as simple, but that's, that's mostly

because the complexity is hidden away or choices have been made on your

behalf to deal with the complexity, but it's still there.

Other systems make other choices and, and still have that, expose

that complexity to you.

I mean, it's a trade-off, right?

Because some people prefer to deal

with the complexity themselves

and not be handheld in that way.

And I find that really interesting how, you know,

that's just a fact of life, right?

Complexity is just there.

It's a matter of how you deal with it, really.

- Yeah, yeah.

And color, there's plenty of complexity in color.

- Oh yeah.

So my next package is called Aoxiang

by the user with a handle isaced, I-S-A-C-E-D.

And Aoxiang is a Chinese term,

apparently translates into saw or hover.

This is a small HTTP web server package.

And I think it's interesting, not just in that space

where you probably use something more heavyweight

than this package, but it's interesting as a prototype

or testing tool.

Imagine you're writing an iOS app

And you need to stand up a service that you can connect to.

For instance, if you're testing on device

and you have no other or better means of injecting

JSON's codable structs, for instance, for connecting to.

And you can stand this up as an endpoint you control.

So typically you might use a Python package, for instance,

which has a couple of tools like this

where you can easily do this.

Or I think Node as well,

Node Express is a term that comes to mind.

I don't have a lot of experience with Node.

But this is sort of the Swift equivalent to that.

And that's really nice.

I tried this, you can actually run it in the playground.

So you can bring this up.

It has a nice API, a closure-based syntax.

So you can write little root handlers.

So it's very easy to stick something in.

And then obviously Swift, as an example,

is very easy because you can use the same

codable structs there that you use in your client app.

That makes it then very easy to stand this up

deploy. It has no dependencies as well, that's nice and that makes it easy to use in in like

scripts or a playground. So that's a package called AUGS-YANG by Aisa Khed. And I should also

mention at this point there was another package that does a very similar thing which is called

Swifter by Damian Kołakowski, which is a package that has been around for quite a while,

that's nine years in development, 99 contributors, which is also a package like this that has no

dependencies and works in a playground or in scripts. So for instance, these are packages

you can use with Swift SH, the Swift runner by MaxHole. So two nice packages for standing up a

little web service for testing and prototyping. ALFIE: I think most languages have something like

this. There's a one-liner that I use with the Ruby executable that will, from one line on the command

line, take a started HTTP server and point it at a set of files in a directory. Now this is obviously

more than that because you can actually define not just serve files but define actual code to execute

when these requests come in. And it's also worth noting that it supports async/await as well,

which is great, so you can fire off async commands on your requests as well.

Yeah, exactly. Yeah, be a nice little package. That's great, and I think that

pretty much brings us to the end of another episode. Thank you so much for

listening everybody, and if there is any feedback or any packages that you

would like us to take a look at on the show, please do drop us a line. The

easiest way to get in contact with us is through our Discord server, which if you

go to the readme file of the main Swift package index server repository, you'll

find a link to the Discord there. So we'd love to have you take part in that. And

we will be back in a couple of weeks with another episode.

Yep, see you in two weeks. Thank you. Bye bye.

All right. Bye bye.