Github/screenshot-to-code
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'main' into hosted

Browse Source
This commit is contained in:
Abi Raja 2024-06-27 17:37:01 +08:00
commit 92ea62e9bf
7 changed files with 118 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
README.md
View File

@ -7,6 +7,7 @@ https://github.com/abi/screenshot-to-code/assets/23818/6cebadae-2fe3-4986-ac6a-8
Supported stacks:
- HTML + Tailwind
- HTML + CSS
- React + Tailwind
- Vue + Tailwind
- Bootstrap
@ -24,7 +25,7 @@ Supported AI models:
See the [Examples](#-examples) section below for more demos.
We also just added experimental support for taking a video/screen recording of a website in action and turning that into a functional prototype.
We also just added experimental support for taking a video/screen recording of a website in action and turning that into a functional prototype.
![google in app quick 3](https://github.com/abi/screenshot-to-code/assets/23818/8758ffa4-9483-4b9b-bb66-abd6d1594c33)
@ -42,12 +43,12 @@ We also just added experimental support for taking a video/screen recording of a
## 🛠 Getting Started
The app has a React/Vite frontend and a FastAPI backend.
The app has a React/Vite frontend and a FastAPI backend.
Keys needed:
* [OpenAI API key with access to GPT-4](https://github.com/abi/screenshot-to-code/blob/main/Troubleshooting.md)
* Anthropic key (optional) - only if you want to use Claude Sonnet, or for experimental video support.
- [OpenAI API key with access to GPT-4](https://github.com/abi/screenshot-to-code/blob/main/Troubleshooting.md)
- Anthropic key (optional) - only if you want to use Claude Sonnet, or for experimental video support.
Run the backend (I use Poetry for package management - `pip install poetry` if you don't have it):
@ -94,9 +95,9 @@ The app will be up and running at http://localhost:5173. Note that you can't dev
- **I'm running into an error when setting up the backend. How can I fix it?** [Try this](https://github.com/abi/screenshot-to-code/issues/3#issuecomment-1814777959). If that still doesn't work, open an issue.
- **How do I get an OpenAI API key?** See https://github.com/abi/screenshot-to-code/blob/main/Troubleshooting.md
- **How can I configure an OpenAI proxy?** - If you're not able to access the OpenAI API directly (due to e.g. country restrictions), you can try a VPN or you can configure the OpenAI base URL to use a proxy: Set OPENAI_BASE_URL in the `backend/.env` or directly in the UI in the settings dialog. Make sure the URL has "v1" in the path so it should look like this: `https://xxx.xxxxx.xxx/v1`
- **How can I configure an OpenAI proxy?** - If you're not able to access the OpenAI API directly (due to e.g. country restrictions), you can try a VPN or you can configure the OpenAI base URL to use a proxy: Set OPENAI_BASE_URL in the `backend/.env` or directly in the UI in the settings dialog. Make sure the URL has "v1" in the path so it should look like this: `https://xxx.xxxxx.xxx/v1`
- **How can I update the backend host that my front-end connects to?** - Configure VITE_HTTP_BACKEND_URL and VITE_WS_BACKEND_URL in front/.env.local For example, set VITE_HTTP_BACKEND_URL=http://124.10.20.1:7001
- **Seeing UTF-8 errors when running the backend?** - On windows, open the .env file with notepad++, then go to Encoding and select UTF-8.
- **Seeing UTF-8 errors when running the backend?** - On windows, open the .env file with notepad++, then go to Encoding and select UTF-8.
- **How can I provide feedback?** For feedback, feature requests and bug reports, open an issue or ping me on [Twitter](https://twitter.com/_abi_).
## 📚 Examples

17
backend/prompts/imported_code_prompts.py
View File

@ -18,6 +18,22 @@ Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
IMPORTED_CODE_HTML_CSS_SYSTEM_PROMPT = """
You are an expert CSS developer.
- Do not add comments in the code such as "<!-- Add other navigation links as needed -->" and "<!-- ... other news items ... -->" in place of writing the full code. WRITE THE FULL CODE.
- Repeat elements as needed. For example, if there are 15 items, the code should have 15 items. DO NOT LEAVE comments like "<!-- Repeat for each news item -->" or bad things will happen.
- For images, use placeholder images from https://placehold.co and include a detailed description of the image in the alt text so that an image generation AI can generate the image later.
In terms of libraries,
- You can use Google Fonts
- Font Awesome for icons: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css"></link>
Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
IMPORTED_CODE_REACT_TAILWIND_SYSTEM_PROMPT = """
You are an expert React/Tailwind developer
@ -128,6 +144,7 @@ Do not include markdown "```" or "```svg" at the start or end.
IMPORTED_CODE_SYSTEM_PROMPTS = SystemPrompts(
html_tailwind=IMPORTED_CODE_TAILWIND_SYSTEM_PROMPT,
html_css=IMPORTED_CODE_HTML_CSS_SYSTEM_PROMPT,
react_tailwind=IMPORTED_CODE_REACT_TAILWIND_SYSTEM_PROMPT,
bootstrap=IMPORTED_CODE_BOOTSTRAP_SYSTEM_PROMPT,
ionic_tailwind=IMPORTED_CODE_IONIC_TAILWIND_SYSTEM_PROMPT,

25
backend/prompts/screenshot_system_prompts.py
View File

@ -26,6 +26,30 @@ Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
HTML_CSS_SYSTEM_PROMPT = """
You are an expert CSS developer
You take screenshots of a reference web page from the user, and then build single page apps
using CSS, HTML and JS.
You might also be given a screenshot(The second image) of a web page that you have already built, and asked to
update it to look more like the reference image(The first image).
- Make sure the app looks exactly like the screenshot.
- Pay close attention to background color, text color, font size, font family,
padding, margin, border, etc. Match the colors and sizes exactly.
- Use the exact text from the screenshot.
- Do not add comments in the code such as "<!-- Add other navigation links as needed -->" and "<!-- ... other news items ... -->" in place of writing the full code. WRITE THE FULL CODE.
- Repeat elements as needed to match the screenshot. For example, if there are 15 items, the code should have 15 items. DO NOT LEAVE comments like "<!-- Repeat for each news item -->" or bad things will happen.
- For images, use placeholder images from https://placehold.co and include a detailed description of the image in the alt text so that an image generation AI can generate the image later.
In terms of libraries,
- You can use Google Fonts
- Font Awesome for icons: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css"></link>
Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
BOOTSTRAP_SYSTEM_PROMPT = """
You are an expert Bootstrap developer
You take screenshots of a reference web page from the user, and then build single page apps
@ -176,6 +200,7 @@ Do not include markdown "```" or "```svg" at the start or end.
SYSTEM_PROMPTS = SystemPrompts(
html_css=HTML_CSS_SYSTEM_PROMPT,
html_tailwind=HTML_TAILWIND_SYSTEM_PROMPT,
react_tailwind=REACT_TAILWIND_SYSTEM_PROMPT,
bootstrap=BOOTSTRAP_SYSTEM_PROMPT,

68
backend/prompts/test_prompts.py
View File

@ -1,3 +1,4 @@
from llm import Llm
from prompts import assemble_imported_code_prompt, assemble_prompt
TAILWIND_SYSTEM_PROMPT = """
@ -25,6 +26,30 @@ Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
HTML_CSS_SYSTEM_PROMPT = """
You are an expert CSS developer
You take screenshots of a reference web page from the user, and then build single page apps
using CSS, HTML and JS.
You might also be given a screenshot(The second image) of a web page that you have already built, and asked to
update it to look more like the reference image(The first image).
- Make sure the app looks exactly like the screenshot.
- Pay close attention to background color, text color, font size, font family,
padding, margin, border, etc. Match the colors and sizes exactly.
- Use the exact text from the screenshot.
- Do not add comments in the code such as "<!-- Add other navigation links as needed -->" and "<!-- ... other news items ... -->" in place of writing the full code. WRITE THE FULL CODE.
- Repeat elements as needed to match the screenshot. For example, if there are 15 items, the code should have 15 items. DO NOT LEAVE comments like "<!-- Repeat for each news item -->" or bad things will happen.
- For images, use placeholder images from https://placehold.co and include a detailed description of the image in the alt text so that an image generation AI can generate the image later.
In terms of libraries,
- You can use Google Fonts
- Font Awesome for icons: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css"></link>
Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
BOOTSTRAP_SYSTEM_PROMPT = """
You are an expert Bootstrap developer
You take screenshots of a reference web page from the user, and then build single page apps
@ -189,6 +214,22 @@ Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
IMPORTED_CODE_HTML_CSS_SYSTEM_PROMPT = """
You are an expert CSS developer.
- Do not add comments in the code such as "<!-- Add other navigation links as needed -->" and "<!-- ... other news items ... -->" in place of writing the full code. WRITE THE FULL CODE.
- Repeat elements as needed. For example, if there are 15 items, the code should have 15 items. DO NOT LEAVE comments like "<!-- Repeat for each news item -->" or bad things will happen.
- For images, use placeholder images from https://placehold.co and include a detailed description of the image in the alt text so that an image generation AI can generate the image later.
In terms of libraries,
- You can use Google Fonts
- Font Awesome for icons: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css"></link>
Return only the full code in <html></html> tags.
Do not include markdown "```" or "```html" at the start or end.
"""
IMPORTED_CODE_REACT_TAILWIND_SYSTEM_PROMPT = """
You are an expert React/Tailwind developer
@ -314,6 +355,12 @@ def test_prompts():
assert tailwind_prompt[0].get("content") == TAILWIND_SYSTEM_PROMPT
assert tailwind_prompt[1]["content"][2]["text"] == USER_PROMPT # type: ignore
html_css_prompt = assemble_prompt(
"image_data_url", "html_css", "result_image_data_url"
)
assert html_css_prompt[0].get("content") == HTML_CSS_SYSTEM_PROMPT
assert html_css_prompt[1]["content"][2]["text"] == USER_PROMPT # type: ignore
react_tailwind_prompt = assemble_prompt(
"image_data_url", "react_tailwind", "result_image_data_url"
)
@ -345,7 +392,7 @@ def test_prompts():
def test_imported_code_prompts():
tailwind_prompt = assemble_imported_code_prompt(
"code", "html_tailwind", "result_image_data_url"
"code", "html_tailwind", Llm.GPT_4O_2024_05_13
)
expected_tailwind_prompt = [
{"role": "system", "content": IMPORTED_CODE_TAILWIND_SYSTEM_PROMPT},
@ -353,8 +400,17 @@ def test_imported_code_prompts():
]
assert tailwind_prompt == expected_tailwind_prompt
html_css_prompt = assemble_imported_code_prompt(
"code", "html_css", Llm.GPT_4O_2024_05_13
)
expected_html_css_prompt = [
{"role": "system", "content": IMPORTED_CODE_HTML_CSS_SYSTEM_PROMPT},
{"role": "user", "content": "Here is the code of the app: code"},
]
assert html_css_prompt == expected_html_css_prompt
react_tailwind_prompt = assemble_imported_code_prompt(
"code", "react_tailwind", "result_image_data_url"
"code", "react_tailwind", Llm.GPT_4O_2024_05_13
)
expected_react_tailwind_prompt = [
{"role": "system", "content": IMPORTED_CODE_REACT_TAILWIND_SYSTEM_PROMPT},
@ -363,7 +419,7 @@ def test_imported_code_prompts():
assert react_tailwind_prompt == expected_react_tailwind_prompt
bootstrap_prompt = assemble_imported_code_prompt(
"code", "bootstrap", "result_image_data_url"
"code", "bootstrap", Llm.GPT_4O_2024_05_13
)
expected_bootstrap_prompt = [
{"role": "system", "content": IMPORTED_CODE_BOOTSTRAP_SYSTEM_PROMPT},
@ -372,7 +428,7 @@ def test_imported_code_prompts():
assert bootstrap_prompt == expected_bootstrap_prompt
ionic_tailwind = assemble_imported_code_prompt(
"code", "ionic_tailwind", "result_image_data_url"
"code", "ionic_tailwind", Llm.GPT_4O_2024_05_13
)
expected_ionic_tailwind = [
{"role": "system", "content": IMPORTED_CODE_IONIC_TAILWIND_SYSTEM_PROMPT},
@ -381,7 +437,7 @@ def test_imported_code_prompts():
assert ionic_tailwind == expected_ionic_tailwind
vue_tailwind = assemble_imported_code_prompt(
"code", "vue_tailwind", "result_image_data_url"
"code", "vue_tailwind", Llm.GPT_4O_2024_05_13
)
expected_vue_tailwind = [
{"role": "system", "content": IMPORTED_CODE_VUE_TAILWIND_PROMPT},
@ -389,7 +445,7 @@ def test_imported_code_prompts():
]
assert vue_tailwind == expected_vue_tailwind
svg = assemble_imported_code_prompt("code", "svg", "result_image_data_url")
svg = assemble_imported_code_prompt("code", "svg", Llm.GPT_4O_2024_05_13)
expected_svg = [
{"role": "system", "content": IMPORTED_CODE_SVG_SYSTEM_PROMPT},
{"role": "user", "content": "Here is the code of the SVG: code"},

2
backend/prompts/types.py
View File

@ -2,6 +2,7 @@ from typing import Literal, TypedDict
class SystemPrompts(TypedDict):
html_css: str
html_tailwind: str
react_tailwind: str
bootstrap: str
@ -11,6 +12,7 @@ class SystemPrompts(TypedDict):
Stack = Literal[
"html_css",
"html_tailwind",
"react_tailwind",
"bootstrap",

3
frontend/src/App.tsx
View File

@ -123,7 +123,8 @@ function App({ navbarComponent }: Props) {
(selectedCodeGenerationModel === CodeGenerationModel.GPT_4O_2024_05_13 ||
selectedCodeGenerationModel ===
CodeGenerationModel.CLAUDE_3_5_SONNET_2024_06_20) &&
settings.generatedCodeConfig === Stack.HTML_TAILWIND;
(settings.generatedCodeConfig === Stack.HTML_TAILWIND ||
settings.generatedCodeConfig === Stack.HTML_CSS);
// Indicate coding state using the browser tab's favicon and title
useBrowserTabIndicator(appState === AppState.CODING);

3
frontend/src/lib/stacks.ts
View File

@ -1,6 +1,8 @@
// Keep in sync with backend (prompts/types.py)
// Order here determines order in dropdown
export enum Stack {
HTML_TAILWIND = "html_tailwind",
HTML_CSS = "html_css",
REACT_TAILWIND = "react_tailwind",
BOOTSTRAP = "bootstrap",
VUE_TAILWIND = "vue_tailwind",
@ -11,6 +13,7 @@ export enum Stack {
export const STACK_DESCRIPTIONS: {
[key in Stack]: { components: string[]; inBeta: boolean };
} = {
html_css: { components: ["HTML", "CSS"], inBeta: false },
html_tailwind: { components: ["HTML", "Tailwind"], inBeta: false },
react_tailwind: { components: ["React", "Tailwind"], inBeta: false },
bootstrap: { components: ["Bootstrap"], inBeta: false },