Tutorial: Multimodal Web Development

As we’re approaching the end of the semester, we’re going to think about how generative AI will influence complex workflows. This week, we’ll build a simple portfolio combining our experiments thus far, and in the process combine new approaches (such as music generation and code conversion) with the workflow we’ve developed over the last few weeks building interactive content for the web.

Combinatorial Prompts

For this exercise, you might find it easiest to build a piece at a time and combine. Try breaking down the elements of the page into small, concrete prompts, and use Visual Studio Code’s tools to assist you in debugging and placement. The goal of this project is to combine image, text, and procedural output from all the approaches we’ve used thus far: select at least four of our exercises to link together in a single website.

You can either create a new GitHub repository or work in one of your existing projects. Remember to distinguish the names of separate .html files, and use the filename index.html for your main page if you decide to build and link multiple pages. Try some combination of the steps below to integrate at least for outputs of text, images, and code together in the website:

• Pull short descriptions from your posts. Use your discussion posts for summaries to accompany each exercise you highlight, contextualizing the output in the conditions of its creation.
• Create containers for multiple images. Consider revisiting projects such as your zine and breaking them into their components: how would you turn them into a simple, web-interactive object? Ask for interfaces that might use techniques like a web carousel or automatically loop through images.
• Embed music and/or video files. Ask about the specific file by name and type (for instance, a generated video in the format “animation.mp4”). Consider asking for customizations in the controls, framing, and in technical settings, like whether the video or music will play automatically.
• Link between different pages. Build an interface, such as a navigation menu, with links to subpages or subsections as preferred. Make sure to watch your file names to avoid overlaps and unintentional deletion of content.
• Convert from Python to JavaScript. Working from either your distant reading output or another of our exercises that combined ChatGPT and Python, take a Python-based visualization and attempt to convert it to a JavaScript-based, web-ready format.

We will review one approach to each of these methods during the live session. Share your page using the same GitHub Pages approach we used in exercises ten and eleven.

Connecting to Modern Workflows

Several new tools and industry betas suggest that software environments are about to change fundamentally: from AI code editor Cursor to GitHub Copilot Workspace and GitHub Spark, there’s been a major investment in the next generation of “text-to-code” alongside less prompt-based interfaces that integrate human expertise. We’ve now seen similar moves across multiple media industries, with implications for publishing, art, game design, software, etc. How do you imagine this process we’ve engaged in will change in the next year? What concerns might Noble’s critiques bring to how we view, and guide, that future?