Implement Jupyter Frontend (#1363)

* initialize plugin definition * initialize plugin definition * simplify mixin * further improve plugin mixin * add cache dir for pip * support clean up cache * add script for setup jupyter and execution server * integrate JupyterRequirement to ssh_box * source bashrc at the end of plugin load * add execute_cli that accept code via stdin * make JUPYTER_EXEC_SERVER_PORT configurable via env var * increase background cmd sleep time * Update opendevin/sandbox/plugins/mixin.py Co-authored-by: Robert Brennan <[email protected]> * add mixin to base class * make jupyter requirement a dataclass * source plugins only when >0 requirements * add `sandbox_plugins` for each agent & have controller take care of it * update build.sh to make logs available in /opendevin/logs * switch to use config for lib and cache dir * Add SANDBOX_WORKSPACE_DIR into config * Add SANDBOX_WORKSPACE_DIR into config * fix occurence of /workspace * fix permission issue with /workspace * use python to implement execute_cli to avoid stdin escape issue * add IPythonRunCellAction and get it working * wait until jupyter is avaialble * support plugin via copying instead of mounting * add agent talk action * support follow-up user language feedback * add __str__ for action to be printed better * only print PLAN at the beginning * wip: update codeact agent * get rid the initial messate * update codeact agent to handle null action; add thought to bash * dispatch thought for RUN action as well * fix weird behavior of pxssh where the output would not flush correctly * make ssh box can handle exit_code properly as well * add initial version of swe-agent plugin; * rename swe cursors * split setup script into two and create two requirements * print SWE-agent command documentation * update swe-agent to default to no custom docs * add initial version of swe-agent plugin; * rename swe cursors * split setup script into two and create two requirements * print SWE-agent command documentation * update swe-agent to default to no custom docs * update dockerfile with dependency from swe-agent * make env setup a separate script for .bashrc source * add wip prompt * fix mount_dir for ssh_box * update prompt * fix mount_dir for ssh_box * default to use host network * default to use host network * move prompt to a separate file * fix swe-tool plugins; add missing _split_string * remove hostname from sshbox * update the prompt with edit functionality * fix swe-tool plugins; add missing _split_string * add awaiting into status bar * fix the bug of additional send event * remove some print action * move logic to config.py * remove debugging comments * make host network as default * make WORKSPACE_MOUNT_PATH as abspath * implement execute_cli via file cp * Revert "implement execute_cli via file cp" This reverts commit 06f0155. * add codeact dependencies to default container * add IPythonRunCellObservation * add back cache dir and default to /tmp * make USE_HOST_NETWORK a bool * revert use host network to false * add temporarily fix for IPython RUN action * preliminary implementation of CodeActAgent's jupyter * update node module * update prompt * revert USE_HOST_NETWORK to true since it is not affecting anything * attempt to fix lint * remove newline * update prompt * Refactor browser style. (#1358) * delete useless assets and css class. * add waiting for page loaded (networkidle with 3s timeout) * Add integration test framework with mock llm (#1301) * Add integration test framework with mock llm * Fix MonologueAgent and PlannerAgent tests * Remove adhoc logging * Use existing logs * Fix SWEAgent and PlannerAgent * Check-in test log files * conftest: look up under test name folder only * Add docstring to conftest * Finish dev doc * Avoid non-determinism * Remove dependency on llm embedding model * Init embedding model only for MonologueAgent * Add adhoc fix for sandbox discrepancy * Test ssh and exec sandboxes * CI: fix missing sandbox type * conftest: Remove hack * Reword comment for TODO * Revert "refactor(frontend): Terminal (#1315)" (#1360) This reverts commit 27246ac. * revert USE_HOST_NETWORK to true since it is not affecting anything * attempt to fix lint * handle IsADirectory errors (#1365) * update to 0.4.0 (#1362) Co-authored-by: Jim Su <[email protected]> * feat(frontend): multiple design changes (#1370) * fix/improve terminal hook (#1371) * Revert "update node module" This reverts commit 459b103. * support SyntaxHighlighter and markdown for jupyter visualization * fix jupyter execution server * make jupyter active * improve the display of markdown and raw text * get base64 image display for react * add `thought` to most action class * fix unit tests for current action abstraction * support user exit * update test cases with the latest action format (added 'thought') * fix integration test for CodeActAGent by mocking stdin * only mock stdin for tests with user_responses.log * remove -exec integration test for CodeActAgent since it is not supported * remove specific stop word * fix comments * improve clarity of prompt * attempt to fix lint * attempt to fix lint yet agiain * fix py lint * fix integration tests * sandbox might failed in chown due to mounting, but it won't be fatal * update debug instruction for sshbox * fix typo * get RUN_AS_DEVIN and network=host working with app sandbox * get RUN_AS_DEVIN and network=host working with app sandbox * attempt to fix the workspace base permission * sandbox might failed in chown due to mounting, but it won't be fatal * update sshbox instruction * remove default user id since it will be passed in the instruction * revert permission fix since it should be resolved by correct SANDBOX_USER_ID * the permission issue can be fixed by simply provide correct env var * remove log * set sandbox user id to getuid by default * move logging to initializer * make the uid consistent across host, app container, and sandbox * remove hostname as it causes sudo issue * fix permission of entrypoint script * make the uvicron app run as host user uid for jupyter plugin * add warning message * fix frontend lint * update dev md for instruction of running unit tests * add back unit tests * revert back to the original sandbox implementation to fix testcases * revert use host network * get docker socket gid and usermod instead of chmod 777 * allow unit test workflow to find docker.sock * make sandbox test working via patch * fix arg parser that's broken for some reason * try to fix app build disk space issue * fix integration test * Revert "fix arg parser that's broken for some reason" This reverts commit 6cc8961. * update Development.md * cleanup intergration tests & add exception for CodeAct+execbox * fix config * implement user_message action * fix doc * fix event dict error * fix frontend lint * revert accidentally changes to integration tests * revert accidentally changes to integration tests --------- Co-authored-by: Robert Brennan <[email protected]> Co-authored-by: Leo <[email protected]> Co-authored-by: Boxuan Li <[email protected]> Co-authored-by: Jim Su <[email protected]> Co-authored-by: Alex Bäuerle <[email protected]> Co-authored-by: sp.wack <[email protected]> Co-authored-by: Robert Brennan <[email protected]>
OpenDevin · May 1, 2024 · d80f025 · d80f025
1 parent 442ab73
commit d80f025
Show file tree

Hide file tree

Showing 12 changed files with 179 additions and 7 deletions.
diff --git a/agenthub/codeact_agent/codeact_agent.py b/agenthub/codeact_agent/codeact_agent.py
@@ -122,6 +122,12 @@ def step(self, state: State) -> Action:
  self.messages.append({'role': 'user', 'content': content})
  elif isinstance(obs, IPythonRunCellObservation):
  content = 'OBSERVATION:\n' + obs.content
+ # replace base64 images with a placeholder
+ splited = content.split('\n')
+ for i, line in enumerate(splited):
+ if '![image](data:image/png;base64,' in line:
+ splited[i] = '![image](data:image/png;base64, ...) already displayed to user'
+ content = '\n'.join(splited)
  self.messages.append({'role': 'user', 'content': content})
  else:
  raise NotImplementedError(

diff --git a/frontend/package-lock.json b/frontend/package-lock.json
diff --git a/frontend/package.json b/frontend/package.json
@@ -25,6 +25,7 @@
  "react": "^18.2.0",
  "react-accessible-treeview": "^2.8.3",
  "react-dom": "^18.2.0",
+ "react-highlight": "^0.15.0",
  "react-hot-toast": "^2.4.1",
  "react-i18next": "^14.1.0",
  "react-icons": "^5.0.1",
@@ -64,6 +65,7 @@
  "@types/node": "^18.0.0 ",
  "@types/react": "^18.2.66",
  "@types/react-dom": "^18.2.22",
+ "@types/react-highlight": "^0.12.8",
  "@types/react-syntax-highlighter": "^15.5.11",
  "@typescript-eslint/eslint-plugin": "^7.4.0",
  "@typescript-eslint/parser": "^7.0.0",

diff --git a/frontend/src/components/Jupyter.tsx b/frontend/src/components/Jupyter.tsx
@@ -0,0 +1,77 @@
+import React from "react";
+import { useSelector } from "react-redux";
+import SyntaxHighlighter from "react-syntax-highlighter";
+import Markdown from "react-markdown";
+import { atomOneDark } from "react-syntax-highlighter/dist/esm/styles/hljs";
+import { RootState } from "#/store";
+import { Cell } from "#/state/jupyterSlice";
+
+interface IJupyterCell {
+ cell: Cell;
+}
+
+function JupyterCell({ cell }: IJupyterCell): JSX.Element {
+ const code = cell.content;
+
+ if (cell.type === "input") {
+ return (
+ <div className="rounded-lg bg-gray-800 dark:bg-gray-900 p-2 text-xs">
+ <div className="mb-1 text-gray-400">EXECUTE</div>
+ <pre
+ className="scrollbar-custom scrollbar-thumb-gray-500 hover:scrollbar-thumb-gray-400 dark:scrollbar-thumb-white/10 dark:hover:scrollbar-thumb-white/20 overflow-auto px-5"
+ style={{ padding: 0, marginBottom: 0, fontSize: "0.75rem" }}
+ >
+ <SyntaxHighlighter language="python" style={atomOneDark}>
+ {code}
+ </SyntaxHighlighter>
+ </pre>
+ </div>
+ );
+ }
+ return (
+ <div className="rounded-lg bg-gray-800 dark:bg-gray-900 p-2 text-xs">
+ <div className="mb-1 text-gray-400">STDOUT/STDERR</div>
+ <pre
+ className="scrollbar-custom scrollbar-thumb-gray-500 hover:scrollbar-thumb-gray-400 dark:scrollbar-thumb-white/10 dark:hover:scrollbar-thumb-white/20 overflow-auto px-5 max-h-[60vh] bg-gray-800"
+ style={{ padding: 0, marginBottom: 0, fontSize: "0.75rem" }}
+ >
+ {/* split code by newline and render each line as a plaintext, except it starts with `![image]` so we render it as markdown */}
+ {code.split("\n").map((line, index) => {
+ if (line.startsWith("![image](data:image/png;base64,")) {
+ // add new line before and after the image
+ return (
+ <div key={index}>
+ <Markdown urlTransform={(value: string) => value}>
+ {line}
+ </Markdown>
+ <br />
+ </div>
+ );
+ }
+ return (
+ <div key={index}>
+ <SyntaxHighlighter language="plaintext" style={atomOneDark}>
+ {line}
+ </SyntaxHighlighter>
+ <br />
+ </div>
+ );
+ })}
+ </pre>
+ </div>
+ );
+}
+
+function Jupyter(): JSX.Element {
+ const { cells } = useSelector((state: RootState) => state.jupyter);
+
+ return (
+ <div className="flex-1 overflow-y-auto flex flex-col">
+ {cells.map((cell, index) => (
+ <JupyterCell key={index} cell={cell} />
+ ))}
+ </div>
+ );
+}
+
+export default Jupyter;
diff --git a/frontend/src/components/Workspace.tsx b/frontend/src/components/Workspace.tsx
@@ -12,6 +12,7 @@ import { AllTabs, TabOption, TabType } from "#/types/TabOption";
 import Browser from "./Browser";
 import CodeEditor from "./CodeEditor";
 import Planner from "./Planner";
+import Jupyter from "./Jupyter";
 
 function Workspace() {
  const { t } = useTranslation();
@@ -20,12 +21,13 @@ function Workspace() {
  const screenshotSrc = useSelector(
  (state: RootState) => state.browser.screenshotSrc,
  );
-
+ const jupyterCells = useSelector((state: RootState) => state.jupyter.cells);
  const [activeTab, setActiveTab] = useState<TabType>(TabOption.CODE);
  const [changes, setChanges] = useState<Record<TabType, boolean>>({
  [TabOption.PLANNER]: false,
  [TabOption.CODE]: false,
  [TabOption.BROWSER]: false,
+ [TabOption.JUPYTER]: false,
  });
 
  const tabData = useMemo(
@@ -45,6 +47,11 @@ function Workspace() {
  icon: <IoIosGlobe size={18} />,
  component: <Browser key="browser" />,
  },
+ [TabOption.JUPYTER]: {
+ name: t(I18nKey.WORKSPACE$JUPYTER_TAB_LABEL),
+ icon: <VscCode size={18} />,
+ component: <Jupyter key="jupyter" />,
+ },
  }),
  [t],
  );
@@ -73,6 +80,14 @@ function Workspace() {
  // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [screenshotSrc]);
 
+ useEffect(() => {
+ if (activeTab !== TabOption.JUPYTER && jupyterCells.length > 0) {
+ // FIXME: This is a temporary solution to show the jupyter tab when the first cell is added
+ // Only need to show the tab only when a cell is added
+ setChanges((prev) => ({ ...prev, [TabOption.JUPYTER]: true }));
+ }
+ }, [jupyterCells]);
+
  return (
  <div className="flex flex-col min-h-0 grow">
  <div

diff --git a/frontend/src/i18n/translation.json b/frontend/src/i18n/translation.json
@@ -39,6 +39,19 @@
  "pt": "Planejador",
  "es": "Planificador"
  },
+ "WORKSPACE$JUPYTER_TAB_LABEL": {
+ "en": "Jupyter IPython",
+ "zh-CN": "Jupyter IPython",
+ "de": "Jupyter IPython",
+ "ko-KR": "Jupyter IPython",
+ "no": "Jupyter IPython",
+ "zh-TW": "Jupyter IPython",
+ "ar": "Jupyter IPython",
+ "fr": "Jupyter IPython",
+ "it": "Jupyter IPython",
+ "pt": "Jupyter IPython",
+ "es": "Jupyter IPython"
+ },
  "WORKSPACE$CODE_EDITOR_TAB_LABEL": {
  "en": "Code Editor",
  "zh-CN": "代码编辑器",

diff --git a/frontend/src/services/actions.ts b/frontend/src/services/actions.ts
@@ -3,6 +3,7 @@ import { setScreenshotSrc, setUrl } from "#/state/browserSlice";
 import { appendAssistantMessage } from "#/state/chatSlice";
 import { setCode, updatePath } from "#/state/codeSlice";
 import { appendInput } from "#/state/commandSlice";
+import { appendJupyterInput } from "#/state/jupyterSlice";
 import { setPlan } from "#/state/planSlice";
 import { setInitialized } from "#/state/taskSlice";
 import store from "#/store";
@@ -45,7 +46,7 @@ const messageActions = {
  if (message.args.thought) {
  store.dispatch(appendAssistantMessage(message.args.thought));
  }
- store.dispatch(appendInput(message.args.code));
+ store.dispatch(appendJupyterInput(message.args.code));
  },
  [ActionType.ADD_TASK]: () => {
  getPlan().then((fetchedPlan) => store.dispatch(setPlan(fetchedPlan)));

diff --git a/frontend/src/services/observations.ts b/frontend/src/services/observations.ts
@@ -3,6 +3,7 @@ import { setUrl, setScreenshotSrc } from "#/state/browserSlice";
 import store from "#/store";
 import { ObservationMessage } from "#/types/Message";
 import { appendOutput } from "#/state/commandSlice";
+import { appendJupyterOutput } from "#/state/jupyterSlice";
 import ObservationType from "#/types/ObservationType";
 
 export function handleObservationMessage(message: ObservationMessage) {
@@ -12,7 +13,7 @@ export function handleObservationMessage(message: ObservationMessage) {
  break;
  case ObservationType.RUN_IPYTHON:
  // FIXME: render this as markdown
- store.dispatch(appendOutput(message.content));
+ store.dispatch(appendJupyterOutput(message.content));
  break;
  case ObservationType.BROWSE:
  if (message.extras?.screenshot) {

diff --git a/frontend/src/state/jupyterSlice.ts b/frontend/src/state/jupyterSlice.ts
@@ -0,0 +1,27 @@
+import { createSlice } from "@reduxjs/toolkit";
+
+export type Cell = {
+ content: string;
+ type: "input" | "output";
+};
+
+const initialCells: Cell[] = [];
+
+export const cellSlice = createSlice({
+ name: "cell",
+ initialState: {
+ cells: initialCells,
+ },
+ reducers: {
+ appendJupyterInput: (state, action) => {
+ state.cells.push({ content: action.payload, type: "input" });
+ },
+ appendJupyterOutput: (state, action) => {
+ state.cells.push({ content: action.payload, type: "output" });
+ },
+ },
+});
+
+export const { appendJupyterInput, appendJupyterOutput } = cellSlice.actions;
+
+export default cellSlice.reducer;
diff --git a/frontend/src/store.ts b/frontend/src/store.ts
@@ -7,6 +7,7 @@ import commandReducer from "./state/commandSlice";
 import errorsReducer from "./state/errorsSlice";
 import planReducer from "./state/planSlice";
 import taskReducer from "./state/taskSlice";
+import jupyterReducer from "./state/jupyterSlice";
 
 export const rootReducer = combineReducers({
  browser: browserReducer,
@@ -17,6 +18,7 @@ export const rootReducer = combineReducers({
  errors: errorsReducer,
  plan: planReducer,
  agent: agentReducer,
+ jupyter: jupyterReducer,
 });
 
 const store = configureStore({

diff --git a/frontend/src/types/TabOption.tsx b/frontend/src/types/TabOption.tsx
@@ -2,10 +2,20 @@ enum TabOption {
  PLANNER = "planner",
  CODE = "code",
  BROWSER = "browser",
+ JUPYTER = "jupyter",
 }
 
-type TabType = TabOption.PLANNER | TabOption.CODE | TabOption.BROWSER;
+type TabType =
+ | TabOption.PLANNER
+ | TabOption.CODE
+ | TabOption.BROWSER
+ | TabOption.JUPYTER;
 
-const AllTabs = [TabOption.CODE, TabOption.BROWSER, TabOption.PLANNER];
+const AllTabs = [
+ TabOption.CODE,
+ TabOption.BROWSER,
+ TabOption.PLANNER,
+ TabOption.JUPYTER,
+];
 
 export { AllTabs, TabOption, type TabType };
diff --git a/opendevin/sandbox/plugins/jupyter/execute_server b/opendevin/sandbox/plugins/jupyter/execute_server
@@ -187,8 +187,7 @@ class JupyterKernel:
  outputs.append(msg['content']['data']['text/plain'])
  if 'image/png' in msg['content']['data']:
  # use markdone to display image (in case of large image)
- # outputs.append(f"\n<img src=\'data:image/png;base64,{msg['content']['data']['image/png']}\'/>\n")
- outputs.append(f"![image](data:image/png;base64,{msg['content']['data']['image/png']})")
+ outputs.append(f"\n![image](data:image/png;base64,{msg['content']['data']['image/png']})\n")
 
  elif msg_type == 'execute_reply':
  execution_done = True