# DeSAPher ## The Problem Working with SAP systems often means dealing with hundreds of tables with cryptic names like MARA, VBAK, or MAKT. Understanding what these tables contain and how they relate to each other traditionally requires extensive documentation lookup or SAP expertise. ## Our Solution We created a tool that automatically fetches and structures SAP table definitions from online documentation. This gives us instant access to: - Field names and their meanings - Data types and lengths - Relationships between tables (via check tables) - Comprehensive descriptions of each field - Table names and descriptions ## Benefits - **Time Saving**: No more manual documentation lookups - **Better Understanding**: Clear visibility of table structures and meanings - **Easier Data Analysis**: Quick reference for field names and their purposes - **Knowledge Sharing**: Makes SAP data structures more accessible to team members ## Future Directions ### 1. Interactive Interface - Build a searchable interface for quick table/field lookups - Implement full-text search across descriptions - Add semantic search using LLMs to find relevant tables by describing needs in plain English ### 2. Visual Data Model - Create interactive graph visualizations showing table relationships - Highlight primary/foreign key connections - Enable visual exploration of the SAP data model ### 3. AI-Powered Data Assistant - Use LLMs with our structured documentation as context - Generate SQL queries from natural language questions - Suggest relevant tables for specific business questions - Provide data model explanations in plain language # Our plan As data scientists working with ERP systems like SAP, we often encounter large datasets with hundreds of cryptically named columns. To better understand these data sources, we aim to web scrape SAP’s data dictionary documentation to create programmatic access to table definitions, column descriptions, and data types. ## Steps 1. Scrape column description for one table (for example, MARA) 2. Bring it in to a easily serachable format 3. Apply process to all tables ### Scrape column description for one table (for example, MARA) ``` python url = 'https://www.sapdatasheet.org/abap/tabl/mara.html' ``` ``` python response = httpx.get(url) response ``` ConnectTimeout: _ssl.c:993: The handshake operation timed out --------------------------------------------------------------------------- ConnectTimeout Traceback (most recent call last) File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:101, in map_httpcore_exceptions()  100 try: --> 101 yield  102 except Exception as exc: File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:250, in HTTPTransport.handle_request(self, request)  249 with map_httpcore_exceptions(): --> 250 resp = self._pool.handle_request(req)  252 assert isinstance(resp.stream, typing.Iterable) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:256, in ConnectionPool.handle_request(self, request)  255 self._close_connections(closing) --> 256 raise exc from None  258 # Return the response. Note that in this case we still have to manage  259 # the point at which the response is closed. File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:236, in ConnectionPool.handle_request(self, request)  234 try:  235 # Send the request on the assigned connection. --> 236 response = connection.handle_request(  237  pool_request.request  238  )  239 except ConnectionNotAvailable:  240 # In some cases a connection may initially be available to  241 # handle a request, but then become unavailable.  242 #  243 # In this case we clear the connection and try again. File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection.py:101, in HTTPConnection.handle_request(self, request)  100 self._connect_failed = True --> 101 raise exc  103 return self._connection.handle_request(request) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection.py:78, in HTTPConnection.handle_request(self, request)  77 if self._connection is None: ---> 78 stream = self._connect(request)  80 ssl_object = stream.get_extra_info("ssl_object") File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection.py:156, in HTTPConnection._connect(self, request)  155 with Trace("start_tls", logger, request, kwargs) as trace: --> 156 stream = stream.start_tls(**kwargs)  157 trace.return_value = stream File /usr/local/lib/python3.12/site-packages/httpcore/_backends/sync.py:154, in SyncStream.start_tls(self, ssl_context, server_hostname, timeout)  150 exc_map: ExceptionMapping = {  151 socket.timeout: ConnectTimeout,  152 OSError: ConnectError,  153 } --> 154 with map_exceptions(exc_map):  155 try: File /usr/local/lib/python3.12/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)  157 try: --> 158 self.gen.throw(value)  159 except StopIteration as exc:  160 # Suppress StopIteration *unless* it's the same exception that  161 # was passed to throw(). This prevents a StopIteration  162 # raised inside the "with" statement from being suppressed. File /usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py:14, in map_exceptions(map)  13 if isinstance(exc, from_exc): ---> 14 raise to_exc(exc) from exc  15 raise ConnectTimeout: _ssl.c:993: The handshake operation timed out The above exception was the direct cause of the following exception: ConnectTimeout Traceback (most recent call last) Cell In[53], line 1 ----> 1 response = httpx.get(url)  2 response File /usr/local/lib/python3.12/site-packages/httpx/_api.py:195, in get(url, params, headers, cookies, auth, proxy, follow_redirects, verify, timeout, trust_env)  174 def get(  175 url: URL | str,  176 *,  (...) 185 trust_env: bool = True,  186 ) -> Response:  187  """  188  Sends a `GET` request.  189  (...) 193  on this function, as `GET` requests should not include a request body.  194  """ --> 195 return request(  196  "GET",  197  url,  198  params=params,  199  headers=headers,  200  cookies=cookies,  201  auth=auth,  202  proxy=proxy,  203  follow_redirects=follow_redirects,  204  verify=verify,  205  timeout=timeout,  206  trust_env=trust_env,  207  ) File /usr/local/lib/python3.12/site-packages/httpx/_api.py:109, in request(method, url, params, content, data, files, json, headers, cookies, auth, proxy, timeout, follow_redirects, verify, trust_env)  57 """  58 Sends an HTTP request.  59  (...) 100 ```  101 """  102 with Client(  103 cookies=cookies,  104 proxy=proxy,  (...) 107 trust_env=trust_env,  108 ) as client: --> 109 return client.request(  110  method=method,  111  url=url,  112  content=content,  113  data=data,  114  files=files,  115  json=json,  116  params=params,  117  headers=headers,  118  auth=auth,  119  follow_redirects=follow_redirects,  120  ) File /usr/local/lib/python3.12/site-packages/httpx/_client.py:825, in Client.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)  810 warnings.warn(message, DeprecationWarning, stacklevel=2)  812 request = self.build_request(  813 method=method,  814 url=url,  (...) 823 extensions=extensions,  824 ) --> 825 return self.send(request, auth=auth, follow_redirects=follow_redirects) File /usr/local/lib/python3.12/site-packages/httpx/_client.py:914, in Client.send(self, request, stream, auth, follow_redirects)  910 self._set_timeout(request)  912 auth = self._build_request_auth(request, auth) --> 914 response = self._send_handling_auth(  915  request,  916  auth=auth,  917  follow_redirects=follow_redirects,  918  history=[],  919 )  920 try:  921 if not stream: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:942, in Client._send_handling_auth(self, request, auth, follow_redirects, history)  939 request = next(auth_flow)  941 while True: --> 942 response = self._send_handling_redirects(  943  request,  944  follow_redirects=follow_redirects,  945  history=history,  946  )  947 try:  948 try: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:979, in Client._send_handling_redirects(self, request, follow_redirects, history)  976 for hook in self._event_hooks["request"]:  977 hook(request) --> 979 response = self._send_single_request(request)  980 try:  981 for hook in self._event_hooks["response"]: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:1014, in Client._send_single_request(self, request)  1009 raise RuntimeError(  1010 "Attempted to send an async request with a sync Client instance."  1011 )  1013 with request_context(request=request): -> 1014 response = transport.handle_request(request)  1016 assert isinstance(response.stream, SyncByteStream)  1018 response.request = request File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:249, in HTTPTransport.handle_request(self, request)  235 import httpcore  237 req = httpcore.Request(  238 method=request.method,  239 url=httpcore.URL(  (...) 247 extensions=request.extensions,  248 ) --> 249 with map_httpcore_exceptions():  250 resp = self._pool.handle_request(req)  252 assert isinstance(resp.stream, typing.Iterable) File /usr/local/lib/python3.12/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)  156 value = typ()  157 try: --> 158 self.gen.throw(value)  159 except StopIteration as exc:  160 # Suppress StopIteration *unless* it's the same exception that  161 # was passed to throw(). This prevents a StopIteration  162 # raised inside the "with" statement from being suppressed.  163 return exc is not value File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:118, in map_httpcore_exceptions()  115 raise  117 message = str(exc) --> 118 raise mapped_exc(message) from exc ConnectTimeout: _ssl.c:993: The handshake operation timed out ``` python sample_text = '''
     
Field Key Data Element Domain Data
Type		 Length Decimal
Places		 Short Description Check
table
1 MANDT MANDT MANDT CLNT 3   0   Client T000
239 FASHGRD FASHGRD FASHGRD CHAR 4   0   Fashion Grade T6WFG
''' ``` ``` python soup = BeautifulSoup(response.text, 'lxml') table = soup.find('table', class_='table table-sm') ``` NameError: name 'response' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[51], line 1 ----> 1 soup = BeautifulSoup(response.text, 'lxml')  2 table = soup.find('table', class_='table table-sm') NameError: name 'response' is not defined ### Bring it in to a easily searchable format ``` python headers = [th.text.strip() for th in table.find('thead').find_all('th')] headers ``` NameError: name 'table' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[17], line 1 ----> 1 headers = [th.text.strip() for th in table.find('thead').find_all('th')]  2 headers NameError: name 'table' is not defined ``` python trs = [tr for tr in table.find('tbody').find_all('tr')] trs[0] ``` NameError: name 'table' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[18], line 1 ----> 1 trs = [tr for tr in table.find('tbody').find_all('tr')]  2 trs[0] NameError: name 'table' is not defined ``` python trs[1] ``` NameError: name 'trs' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[19], line 1 ----> 1 trs[1] NameError: name 'trs' is not defined ``` python [td.text.strip() for td in trs[1].find_all('td')] ``` NameError: name 'trs' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[20], line 1 ----> 1 [td.text.strip() for td in trs[1].find_all('td')] NameError: name 'trs' is not defined ``` python rows = [] for tr in table.find('tbody').find_all('tr'): row = [td.text.strip() for td in tr.find_all('td')] rows.append(row) rows[10] ``` NameError: name 'table' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[21], line 2  1 rows = [] ----> 2 for tr in table.find('tbody').find_all('tr'):  3 row = [td.text.strip() for td in tr.find_all('td')]  4 rows.append(row)  5 NameError: name 'table' is not defined ``` python df = pd.DataFrame(rows, columns=headers) df.head() ``` NameError: name 'headers' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[22], line 1 ----> 1 df = pd.DataFrame(rows, columns=headers)  2 df.head() NameError: name 'headers' is not defined ------------------------------------------------------------------------ source ### get_sap_table_structure ``` python def get_sap_table_structure( url ): ``` *Scrapes SAP table structure from sapdatasheet.org and returns a pandas DataFrame* ``` python df = get_sap_table_structure(url) df.head() ``` ReadTimeout: The read operation timed out --------------------------------------------------------------------------- ReadTimeout Traceback (most recent call last) File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:101, in map_httpcore_exceptions()  100 try: --> 101 yield  102 except Exception as exc: File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:250, in HTTPTransport.handle_request(self, request)  249 with map_httpcore_exceptions(): --> 250 resp = self._pool.handle_request(req)  252 assert isinstance(resp.stream, typing.Iterable) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:256, in ConnectionPool.handle_request(self, request)  255 self._close_connections(closing) --> 256 raise exc from None  258 # Return the response. Note that in this case we still have to manage  259 # the point at which the response is closed. File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:236, in ConnectionPool.handle_request(self, request)  234 try:  235 # Send the request on the assigned connection. --> 236 response = connection.handle_request(  237  pool_request.request  238  )  239 except ConnectionNotAvailable:  240 # In some cases a connection may initially be available to  241 # handle a request, but then become unavailable.  242 #  243 # In this case we clear the connection and try again. File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection.py:103, in HTTPConnection.handle_request(self, request)  101 raise exc --> 103 return self._connection.handle_request(request) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:136, in HTTP11Connection.handle_request(self, request)  135 self._response_closed() --> 136 raise exc File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:106, in HTTP11Connection.handle_request(self, request)  97 with Trace(  98 "receive_response_headers", logger, request, kwargs  99 ) as trace:  100 (  101 http_version,  102 status,  103 reason_phrase,  104 headers,  105 trailing_data, --> 106 ) = self._receive_response_headers(**kwargs)  107 trace.return_value = (  108 http_version,  109 status,  110 reason_phrase,  111 headers,  112 ) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:177, in HTTP11Connection._receive_response_headers(self, request)  176 while True: --> 177 event = self._receive_event(timeout=timeout)  178 if isinstance(event, h11.Response): File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:217, in HTTP11Connection._receive_event(self, timeout)  216 if event is h11.NEED_DATA: --> 217 data = self._network_stream.read(  218  self.READ_NUM_BYTES, timeout=timeout  219  )  221 # If we feed this case through h11 we'll raise an exception like:  222 #  223 # httpcore.RemoteProtocolError: can't handle event type  (...) 227 # perspective. Instead we handle this case distinctly and treat  228 # it as a ConnectError. File /usr/local/lib/python3.12/site-packages/httpcore/_backends/sync.py:126, in SyncStream.read(self, max_bytes, timeout)  125 exc_map: ExceptionMapping = {socket.timeout: ReadTimeout, OSError: ReadError} --> 126 with map_exceptions(exc_map):  127 self._sock.settimeout(timeout) File /usr/local/lib/python3.12/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)  157 try: --> 158 self.gen.throw(value)  159 except StopIteration as exc:  160 # Suppress StopIteration *unless* it's the same exception that  161 # was passed to throw(). This prevents a StopIteration  162 # raised inside the "with" statement from being suppressed. File /usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py:14, in map_exceptions(map)  13 if isinstance(exc, from_exc): ---> 14 raise to_exc(exc) from exc  15 raise ReadTimeout: The read operation timed out The above exception was the direct cause of the following exception: ReadTimeout Traceback (most recent call last) Cell In[24], line 1 ----> 1 df = get_sap_table_structure(url)  2 df.head() Cell In[23], line 9, in get_sap_table_structure(url)  5 import httpx  6 from bs4 import BeautifulSoup  7 import pandas as pd  8 ----> 9 response = httpx.get(url)  10 soup = BeautifulSoup(response.text, 'lxml')  11  12 table = soup.find('table', class_='table table-sm') File /usr/local/lib/python3.12/site-packages/httpx/_api.py:195, in get(url, params, headers, cookies, auth, proxy, follow_redirects, verify, timeout, trust_env)  174 def get(  175 url: URL | str,  176 *,  (...) 185 trust_env: bool = True,  186 ) -> Response:  187  """  188  Sends a `GET` request.  189  (...) 193  on this function, as `GET` requests should not include a request body.  194  """ --> 195 return request(  196  "GET",  197  url,  198  params=params,  199  headers=headers,  200  cookies=cookies,  201  auth=auth,  202  proxy=proxy,  203  follow_redirects=follow_redirects,  204  verify=verify,  205  timeout=timeout,  206  trust_env=trust_env,  207  ) File /usr/local/lib/python3.12/site-packages/httpx/_api.py:109, in request(method, url, params, content, data, files, json, headers, cookies, auth, proxy, timeout, follow_redirects, verify, trust_env)  57 """  58 Sends an HTTP request.  59  (...) 100 ```  101 """  102 with Client(  103 cookies=cookies,  104 proxy=proxy,  (...) 107 trust_env=trust_env,  108 ) as client: --> 109 return client.request(  110  method=method,  111  url=url,  112  content=content,  113  data=data,  114  files=files,  115  json=json,  116  params=params,  117  headers=headers,  118  auth=auth,  119  follow_redirects=follow_redirects,  120  ) File /usr/local/lib/python3.12/site-packages/httpx/_client.py:825, in Client.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)  810 warnings.warn(message, DeprecationWarning, stacklevel=2)  812 request = self.build_request(  813 method=method,  814 url=url,  (...) 823 extensions=extensions,  824 ) --> 825 return self.send(request, auth=auth, follow_redirects=follow_redirects) File /usr/local/lib/python3.12/site-packages/httpx/_client.py:914, in Client.send(self, request, stream, auth, follow_redirects)  910 self._set_timeout(request)  912 auth = self._build_request_auth(request, auth) --> 914 response = self._send_handling_auth(  915  request,  916  auth=auth,  917  follow_redirects=follow_redirects,  918  history=[],  919 )  920 try:  921 if not stream: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:942, in Client._send_handling_auth(self, request, auth, follow_redirects, history)  939 request = next(auth_flow)  941 while True: --> 942 response = self._send_handling_redirects(  943  request,  944  follow_redirects=follow_redirects,  945  history=history,  946  )  947 try:  948 try: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:979, in Client._send_handling_redirects(self, request, follow_redirects, history)  976 for hook in self._event_hooks["request"]:  977 hook(request) --> 979 response = self._send_single_request(request)  980 try:  981 for hook in self._event_hooks["response"]: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:1014, in Client._send_single_request(self, request)  1009 raise RuntimeError(  1010 "Attempted to send an async request with a sync Client instance."  1011 )  1013 with request_context(request=request): -> 1014 response = transport.handle_request(request)  1016 assert isinstance(response.stream, SyncByteStream)  1018 response.request = request File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:249, in HTTPTransport.handle_request(self, request)  235 import httpcore  237 req = httpcore.Request(  238 method=request.method,  239 url=httpcore.URL(  (...) 247 extensions=request.extensions,  248 ) --> 249 with map_httpcore_exceptions():  250 resp = self._pool.handle_request(req)  252 assert isinstance(resp.stream, typing.Iterable) File /usr/local/lib/python3.12/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)  156 value = typ()  157 try: --> 158 self.gen.throw(value)  159 except StopIteration as exc:  160 # Suppress StopIteration *unless* it's the same exception that  161 # was passed to throw(). This prevents a StopIteration  162 # raised inside the "with" statement from being suppressed.  163 return exc is not value File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:118, in map_httpcore_exceptions()  115 raise  117 message = str(exc) --> 118 raise mapped_exc(message) from exc ReadTimeout: The read operation timed out ``` python url_2 = 'https://www.sapdatasheet.org/abap/tabl/makt.html' df_2 = get_sap_table_structure(url_2) df_2 ``` ReadTimeout: The read operation timed out --------------------------------------------------------------------------- ReadTimeout Traceback (most recent call last) File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:101, in map_httpcore_exceptions()  100 try: --> 101 yield  102 except Exception as exc: File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:250, in HTTPTransport.handle_request(self, request)  249 with map_httpcore_exceptions(): --> 250 resp = self._pool.handle_request(req)  252 assert isinstance(resp.stream, typing.Iterable) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:256, in ConnectionPool.handle_request(self, request)  255 self._close_connections(closing) --> 256 raise exc from None  258 # Return the response. Note that in this case we still have to manage  259 # the point at which the response is closed. File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:236, in ConnectionPool.handle_request(self, request)  234 try:  235 # Send the request on the assigned connection. --> 236 response = connection.handle_request(  237  pool_request.request  238  )  239 except ConnectionNotAvailable:  240 # In some cases a connection may initially be available to  241 # handle a request, but then become unavailable.  242 #  243 # In this case we clear the connection and try again. File /usr/local/lib/python3.12/site-packages/httpcore/_sync/connection.py:103, in HTTPConnection.handle_request(self, request)  101 raise exc --> 103 return self._connection.handle_request(request) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:136, in HTTP11Connection.handle_request(self, request)  135 self._response_closed() --> 136 raise exc File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:106, in HTTP11Connection.handle_request(self, request)  97 with Trace(  98 "receive_response_headers", logger, request, kwargs  99 ) as trace:  100 (  101 http_version,  102 status,  103 reason_phrase,  104 headers,  105 trailing_data, --> 106 ) = self._receive_response_headers(**kwargs)  107 trace.return_value = (  108 http_version,  109 status,  110 reason_phrase,  111 headers,  112 ) File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:177, in HTTP11Connection._receive_response_headers(self, request)  176 while True: --> 177 event = self._receive_event(timeout=timeout)  178 if isinstance(event, h11.Response): File /usr/local/lib/python3.12/site-packages/httpcore/_sync/http11.py:217, in HTTP11Connection._receive_event(self, timeout)  216 if event is h11.NEED_DATA: --> 217 data = self._network_stream.read(  218  self.READ_NUM_BYTES, timeout=timeout  219  )  221 # If we feed this case through h11 we'll raise an exception like:  222 #  223 # httpcore.RemoteProtocolError: can't handle event type  (...) 227 # perspective. Instead we handle this case distinctly and treat  228 # it as a ConnectError. File /usr/local/lib/python3.12/site-packages/httpcore/_backends/sync.py:126, in SyncStream.read(self, max_bytes, timeout)  125 exc_map: ExceptionMapping = {socket.timeout: ReadTimeout, OSError: ReadError} --> 126 with map_exceptions(exc_map):  127 self._sock.settimeout(timeout) File /usr/local/lib/python3.12/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)  157 try: --> 158 self.gen.throw(value)  159 except StopIteration as exc:  160 # Suppress StopIteration *unless* it's the same exception that  161 # was passed to throw(). This prevents a StopIteration  162 # raised inside the "with" statement from being suppressed. File /usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py:14, in map_exceptions(map)  13 if isinstance(exc, from_exc): ---> 14 raise to_exc(exc) from exc  15 raise ReadTimeout: The read operation timed out The above exception was the direct cause of the following exception: ReadTimeout Traceback (most recent call last) Cell In[25], line 3  1 url_2 = 'https://www.sapdatasheet.org/abap/tabl/makt.html'  2 ----> 3 df_2 = get_sap_table_structure(url_2)  4 df_2 Cell In[23], line 9, in get_sap_table_structure(url)  5 import httpx  6 from bs4 import BeautifulSoup  7 import pandas as pd  8 ----> 9 response = httpx.get(url)  10 soup = BeautifulSoup(response.text, 'lxml')  11  12 table = soup.find('table', class_='table table-sm') File /usr/local/lib/python3.12/site-packages/httpx/_api.py:195, in get(url, params, headers, cookies, auth, proxy, follow_redirects, verify, timeout, trust_env)  174 def get(  175 url: URL | str,  176 *,  (...) 185 trust_env: bool = True,  186 ) -> Response:  187  """  188  Sends a `GET` request.  189  (...) 193  on this function, as `GET` requests should not include a request body.  194  """ --> 195 return request(  196  "GET",  197  url,  198  params=params,  199  headers=headers,  200  cookies=cookies,  201  auth=auth,  202  proxy=proxy,  203  follow_redirects=follow_redirects,  204  verify=verify,  205  timeout=timeout,  206  trust_env=trust_env,  207  ) File /usr/local/lib/python3.12/site-packages/httpx/_api.py:109, in request(method, url, params, content, data, files, json, headers, cookies, auth, proxy, timeout, follow_redirects, verify, trust_env)  57 """  58 Sends an HTTP request.  59  (...) 100 ```  101 """  102 with Client(  103 cookies=cookies,  104 proxy=proxy,  (...) 107 trust_env=trust_env,  108 ) as client: --> 109 return client.request(  110  method=method,  111  url=url,  112  content=content,  113  data=data,  114  files=files,  115  json=json,  116  params=params,  117  headers=headers,  118  auth=auth,  119  follow_redirects=follow_redirects,  120  ) File /usr/local/lib/python3.12/site-packages/httpx/_client.py:825, in Client.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)  810 warnings.warn(message, DeprecationWarning, stacklevel=2)  812 request = self.build_request(  813 method=method,  814 url=url,  (...) 823 extensions=extensions,  824 ) --> 825 return self.send(request, auth=auth, follow_redirects=follow_redirects) File /usr/local/lib/python3.12/site-packages/httpx/_client.py:914, in Client.send(self, request, stream, auth, follow_redirects)  910 self._set_timeout(request)  912 auth = self._build_request_auth(request, auth) --> 914 response = self._send_handling_auth(  915  request,  916  auth=auth,  917  follow_redirects=follow_redirects,  918  history=[],  919 )  920 try:  921 if not stream: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:942, in Client._send_handling_auth(self, request, auth, follow_redirects, history)  939 request = next(auth_flow)  941 while True: --> 942 response = self._send_handling_redirects(  943  request,  944  follow_redirects=follow_redirects,  945  history=history,  946  )  947 try:  948 try: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:979, in Client._send_handling_redirects(self, request, follow_redirects, history)  976 for hook in self._event_hooks["request"]:  977 hook(request) --> 979 response = self._send_single_request(request)  980 try:  981 for hook in self._event_hooks["response"]: File /usr/local/lib/python3.12/site-packages/httpx/_client.py:1014, in Client._send_single_request(self, request)  1009 raise RuntimeError(  1010 "Attempted to send an async request with a sync Client instance."  1011 )  1013 with request_context(request=request): -> 1014 response = transport.handle_request(request)  1016 assert isinstance(response.stream, SyncByteStream)  1018 response.request = request File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:249, in HTTPTransport.handle_request(self, request)  235 import httpcore  237 req = httpcore.Request(  238 method=request.method,  239 url=httpcore.URL(  (...) 247 extensions=request.extensions,  248 ) --> 249 with map_httpcore_exceptions():  250 resp = self._pool.handle_request(req)  252 assert isinstance(resp.stream, typing.Iterable) File /usr/local/lib/python3.12/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)  156 value = typ()  157 try: --> 158 self.gen.throw(value)  159 except StopIteration as exc:  160 # Suppress StopIteration *unless* it's the same exception that  161 # was passed to throw(). This prevents a StopIteration  162 # raised inside the "with" statement from being suppressed.  163 return exc is not value File /usr/local/lib/python3.12/site-packages/httpx/_transports/default.py:118, in map_httpcore_exceptions()  115 raise  117 message = str(exc) --> 118 raise mapped_exc(message) from exc ReadTimeout: The read operation timed out ### Apply process to all tables ------------------------------------------------------------------------ source ### get_sap_table_url ``` python def get_sap_table_url( table_name ): ``` *Constructs sapdatasheet.org URL from SAP table name* ``` python get_sap_table_url('VBAK') ``` 'https://www.sapdatasheet.org/abap/tabl/vbak.html' ------------------------------------------------------------------------ source ### get_sap_tables_structure ``` python def get_sap_tables_structure( tables ): ``` *Gets structure for multiple SAP tables and combines them into one DataFrame with a column indicating the source table* ------------------------------------------------------------------------ source ### get_sap_table_structure ``` python def get_sap_table_structure( url ): ``` *Scrapes SAP table structure from sapdatasheet.org and returns a pandas DataFrame* Returns None if table not found or other error occurs ------------------------------------------------------------------------ source ### get_sap_tables_structure ``` python def get_sap_tables_structure( tables ): ``` *Gets structure for multiple SAP tables and combines them into one DataFrame with a column indicating the source table* ``` python tables = ['MARC', 'MARD', 'MARM', 'MBEW'] ``` ``` python df = get_sap_tables_structure(tables) df.sample(10) ``` Error processing https://www.sapdatasheet.org/abap/tabl/marc.html: The read operation timed out Error processing https://www.sapdatasheet.org/abap/tabl/mard.html: The read operation timed out Error processing https://www.sapdatasheet.org/abap/tabl/marm.html: The read operation timed out Error processing https://www.sapdatasheet.org/abap/tabl/mbew.html: The read operation timed out AttributeError: 'NoneType' object has no attribute 'sample' --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[32], line 2  1 df = get_sap_tables_structure(tables) ----> 2 df.sample(10) AttributeError: 'NoneType' object has no attribute 'sample' ### Let’s also grab the table description for each table. ``` python url ``` 'https://www.sapdatasheet.org/abap/tabl/mara.html' ------------------------------------------------------------------------ source ### get_sap_table_description ``` python def get_sap_table_description( url ): ``` *Scrapes SAP table description from sapdatasheet.org* Returns None if not found or error occurs ``` python get_sap_table_description(url) ``` Error getting description from https://www.sapdatasheet.org/abap/tabl/mara.html: Server error '500 Internal Server Error' for url 'https://www.sapdatasheet.org/abap/tabl/mara.html' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500 ``` python from pathlib import Path ``` ``` python CACHE_PATH = Path('sap_sheet.parquet') ``` ``` python def get_sap_tables_structure(tables, cache_path=CACHE_PATH, force_refresh=False): """ Gets structure for multiple SAP tables, caching results in a single parquet file. Only scrapes tables not already in the cache. force_refresh=True re-scrapes everything. """ if force_refresh and cache_path.exists(): cache_path.unlink() # Load existing cache if cache_path.exists(): cached = pd.read_parquet(cache_path) already_cached = set(cached['Table'].unique()) else: cached = None already_cached = set() # Scrape only missing tables missing = [t for t in tables if t.upper() not in already_cached] new_dfs = [] for table in missing: url = get_sap_table_url(table) df = get_sap_table_structure(url) if df is not None: df['Table'] = table.upper() df['Table Description'] = get_sap_table_description(url) new_dfs.append(df) # Combine and save all_dfs = [d for d in [cached] + new_dfs if d is not None] if not all_dfs: return None result = pd.concat(all_dfs, ignore_index=True) if missing: # only write if something new was added result.to_parquet(cache_path, index=False) # Return only requested tables return result[result['Table'].isin([t.upper() for t in tables])] ``` ``` python sap_sheet = get_sap_tables_structure(tables) sap_sheet.sample(10) ``` Error processing https://www.sapdatasheet.org/abap/tabl/marc.html: The read operation timed out Error processing https://www.sapdatasheet.org/abap/tabl/mard.html: The read operation timed out Error processing https://www.sapdatasheet.org/abap/tabl/marm.html: _ssl.c:993: The handshake operation timed out Error processing https://www.sapdatasheet.org/abap/tabl/mbew.html: _ssl.c:993: The handshake operation timed out AttributeError: 'NoneType' object has no attribute 'sample' --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[39], line 2  1 sap_sheet = get_sap_tables_structure(tables) ----> 2 sap_sheet.sample(10) AttributeError: 'NoneType' object has no attribute 'sample' ``` python sap_sheet['Table Description'].unique() ``` TypeError: 'NoneType' object is not subscriptable --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[40], line 1 ----> 1 sap_sheet['Table Description'].unique() TypeError: 'NoneType' object is not subscriptable ``` python data = { 'MANDT': [100, 100, 100], 'MATNR': ['MAT001', 'MAT002', 'MAT003'], 'WERKS': ['1000', '1000', '2000'], 'PSTAT': ['KVEB', 'KVEB', 'KVEB'], 'LVORM': [None, None, None], 'BWTTY': [None, None, None], 'MMSTA': ['1', '1', '9'], 'MMSTD': [20200101, 20200115, 20210201], 'MAABC': ['A', 'B', None], 'KZKRI': [None, None, None], 'EKGRP': ['100', '100', '200'], 'DISPO': ['001', '001', '002'], 'BESKZ': ['E', 'E', 'X'], 'SOBSL': [None, None, None], 'EISBE': [10.0, 20.0, 15.0], 'MABST': [100.0, 200.0, 150.0], 'ALTSL': [None, None, None], 'KZAUS': [None, None, None], 'AUSDT': [0, 0, 0], 'NFMAT': [None, None, None], 'KZBED': [None, 'T', None], 'RGEKZ': [None, None, None], 'FEVOR': ['G01', None, 'G02'], 'BASMG': [1.0, 1.0, 1.0], 'STAWN': ['84141025', '84141025', '84148073'], 'HERKL': ['DE', 'DE', 'US'], 'HERKR': ['05', '05', '16'], 'EXPME': ['ST', 'ST', 'ST'], 'MTVER': ['1', '1', '1'], 'PRCTR': ['PC100', 'PC100', 'PC200'], 'VERKZ': [None, 'X', None], 'STLAL': [None, None, None], 'STLAN': [None, None, None], 'PLNNR': [None, None, None], 'APLAL': [None, None, None], 'FRTME': [None, None, None], 'LGPRO': ['1001', '1001', '2001'], 'DISGR': ['2000', '2000', '6000'], 'SERNP': [None, None, None], 'PREFE': [None, None, None], 'PRENE': [None, None, None], 'SCHGT': [None, None, None], 'MCRUE': ['X', 'X', 'X'], 'LFGJA': [2024, 2024, 2025], 'EISLO': [0.0, 0.0, 0.0], 'TARGET_STOCK': [50.0, 100.0, 75.0], 'SCM_SCOST': [0.0, 0.0, 0.0], 'SCM_LSUOM': [None, None, None], 'SCM_STRA1': [None, None, None], } df_marc = pd.DataFrame(data) df_marc ```
MANDT MATNR WERKS PSTAT LVORM BWTTY MMSTA MMSTD MAABC KZKRI ... PREFE PRENE SCHGT MCRUE LFGJA EISLO TARGET_STOCK SCM_SCOST SCM_LSUOM SCM_STRA1
0 100 MAT001 1000 KVEB None None 1 20200101 A None ... None None None X 2024 0.0 50.0 0.0 None None
1 100 MAT002 1000 KVEB None None 1 20200115 B None ... None None None X 2024 0.0 100.0 0.0 None None
2 100 MAT003 2000 KVEB None None 9 20210201 None None ... None None None X 2025 0.0 75.0 0.0 None None

3 rows × 49 columns

------------------------------------------------------------------------ source ### convert_sap_types ``` python def convert_sap_types( df:DataFrame, sap_sheet:DataFrame )->DataFrame: ``` *Converts the columns of the ‘df’ DataFrame to the correct data types based on ‘sap_sheet’.* :param df: Pandas DataFrame containing SAP table data (e.g., “df”). :param sap_sheet: Pandas DataFrame containing SAP metadata with column data types. :return: Converted Pandas DataFrame. ``` python df_marc.info() ``` RangeIndex: 3 entries, 0 to 2 Data columns (total 49 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 MANDT 3 non-null int64 1 MATNR 3 non-null object 2 WERKS 3 non-null object 3 PSTAT 3 non-null object 4 LVORM 0 non-null object 5 BWTTY 0 non-null object 6 MMSTA 3 non-null object 7 MMSTD 3 non-null int64 8 MAABC 2 non-null object 9 KZKRI 0 non-null object 10 EKGRP 3 non-null object 11 DISPO 3 non-null object 12 BESKZ 3 non-null object 13 SOBSL 0 non-null object 14 EISBE 3 non-null float64 15 MABST 3 non-null float64 16 ALTSL 0 non-null object 17 KZAUS 0 non-null object 18 AUSDT 3 non-null int64 19 NFMAT 0 non-null object 20 KZBED 1 non-null object 21 RGEKZ 0 non-null object 22 FEVOR 2 non-null object 23 BASMG 3 non-null float64 24 STAWN 3 non-null object 25 HERKL 3 non-null object 26 HERKR 3 non-null object 27 EXPME 3 non-null object 28 MTVER 3 non-null object 29 PRCTR 3 non-null object 30 VERKZ 1 non-null object 31 STLAL 0 non-null object 32 STLAN 0 non-null object 33 PLNNR 0 non-null object 34 APLAL 0 non-null object 35 FRTME 0 non-null object 36 LGPRO 3 non-null object 37 DISGR 3 non-null object 38 SERNP 0 non-null object 39 PREFE 0 non-null object 40 PRENE 0 non-null object 41 SCHGT 0 non-null object 42 MCRUE 3 non-null object 43 LFGJA 3 non-null int64 44 EISLO 3 non-null float64 45 TARGET_STOCK 3 non-null float64 46 SCM_SCOST 3 non-null float64 47 SCM_LSUOM 0 non-null object 48 SCM_STRA1 0 non-null object dtypes: float64(6), int64(4), object(39) memory usage: 1.3+ KB ``` python df_marc_converted = convert_sap_types(df_marc, sap_sheet) df_marc_converted.head() ``` AttributeError: 'NoneType' object has no attribute 'rename' --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[45], line 1 ----> 1 df_marc_converted = convert_sap_types(df_marc, sap_sheet)  2 df_marc_converted.head() Cell In[43], line 9, in convert_sap_types(df, sap_sheet)  5 :param df: Pandas DataFrame containing SAP table data (e.g., "df").  6 :param sap_sheet: Pandas DataFrame containing SAP metadata with column data types.  7 :return: Converted Pandas DataFrame.  8 """ ----> 9 sap_sheet = sap_sheet.rename(columns=lambda x: x.strip())  10 column_type_mapping = sap_sheet.set_index("Field")["DataType"].to_dict()  11  12 for column, sap_type in column_type_mapping.items(): AttributeError: 'NoneType' object has no attribute 'rename' ``` python df_marc_converted.info() ``` NameError: name 'df_marc_converted' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[46], line 1 ----> 1 df_marc_converted.info() NameError: name 'df_marc_converted' is not defined ------------------------------------------------------------------------ source ### rename_sap_columns ``` python def rename_sap_columns( df:DataFrame, sap_sheet:DataFrame )->DataFrame: ``` *Renames the columns in the ‘df’ DataFrame using the ‘Short Description’ from ‘sap_sheet’.* If a column is not found in ‘sap_sheet’, it remains unchanged. :param df: Pandas DataFrame containing SAP table data (e.g., “df”). :param sap_sheet: Pandas DataFrame containing SAP metadata with column names and short descriptions. :return: DataFrame with renamed columns. ``` python df_marc_converted = rename_sap_columns(df_marc_converted, sap_sheet) df_marc_converted.info() ``` NameError: name 'df_marc_converted' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[48], line 1 ----> 1 df_marc_converted = rename_sap_columns(df_marc_converted, sap_sheet)  2 df_marc_converted.info() NameError: name 'df_marc_converted' is not defined ``` python df_marc_converted = clean_col_names(df_marc_converted) df_marc_converted.info() ``` NameError: name 'df_marc_converted' is not defined --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[49], line 1 ----> 1 df_marc_converted = clean_col_names(df_marc_converted)  2 df_marc_converted.info() NameError: name 'df_marc_converted' is not defined # Next steps - we should write some kind of setup functionality to scrape the sap_sheet once and the have it as a lookup reference. - We have duplicates in the `Field` column, because columns are in multiple tables. - We also have duplicates in the `Short Description` column, sometimes even when the `Field` value isn’t the same. - We could combine columns, for example `Field`, `Table`, `Short Description`, which would make a unique value. But currently we only match over the `Field` column. Therefore we can not decide which description we should take. We either have to set a sensible default or we have to have a way to know where the column originated from. The second option seem only viable if we just process sap columns from one sap table. - the sensible default could be based on the number of times the table is used in other tables (the more usages the better). ok, how to best cache the data to not always scrape the data at each run ##### 🤖Reply🤖 A few good options, from simplest to most robust: 1. **Save to Parquet/CSV** — scrape once, save to file, load from file on subsequent runs. Parquet is nice because it preserves dtypes. 2. **Pickle the DataFrame** — similar but keeps all Python types exactly. 3. **SQLite** — good if you want to query the data without loading it all into memory. For your use case, I’d recommend **Parquet** — it’s compact, fast, preserves types well, and you can easily inspect it with pandas. The pattern would be: check if the file exists → if yes, load it; if not, scrape and save. Something like: ``` python from pathlib import Path CACHE_PATH = Path('sap_sheet.parquet') def get_or_scrape_sap_tables(tables, cache_path=CACHE_PATH): if cache_path.exists(): return pd.read_parquet(cache_path) df = get_sap_tables_structure(tables) df.to_parquet(cache_path, index=False) return df ``` You could also add a `force_refresh=False` parameter to re-scrape when needed. Want to go with this approach, or would you prefer something different? what if it is not the same tables every time? ##### 🤖Reply🤖 Good point! The simplest solution is to **cache per table** — one file per SAP table. Then you only scrape tables you haven’t seen before: ``` python CACHE_DIR = Path('sap_cache') CACHE_DIR.mkdir(exist_ok=True) def get_or_scrape_sap_table(table): cache_file = CACHE_DIR / f"{table.upper()}.parquet" if cache_file.exists(): return pd.read_parquet(cache_file) df = get_sap_table_structure(get_sap_table_url(table)) if df is not None: df['Table'] = table df['Table Description'] = get_sap_table_description(get_sap_table_url(table)) df.to_parquet(cache_file, index=False) return df ``` Then [`get_sap_tables_structure`](https://MIS-Analytics.github.io/mis_analytics/desapher.html#get_sap_tables_structure) just calls this per table and concatenates. This way any previously scraped table is free, and only new ones hit the web. Would you like to wire this into [`get_sap_tables_structure`](https://MIS-Analytics.github.io/mis_analytics/desapher.html#get_sap_tables_structure), or would you prefer to try that yourself? maybe there is a middle ground where we only have one parquet file which has all the tables downloaded so far. then new ones are appended and a force_refresh deletes and recreates the file ##### 🤖Reply🤖 That’s a nice middle ground! The logic would be: 1. If cache exists, load it and check which of the requested tables are already in it 2. Scrape only the missing ones 3. Append and save back 4. `force_refresh=True` nukes the file and re-scrapes everything Want to try writing that yourself, or would you like me to write it out? write it for me ##### 🤖Reply🤖 ``` python CACHE_PATH = Path('sap_sheet.parquet') def get_sap_tables_structure(tables, cache_path=CACHE_PATH, force_refresh=False): """ Gets structure for multiple SAP tables, caching results in a single parquet file. Only scrapes tables not already in the cache. force_refresh=True re-scrapes everything. """ if force_refresh and cache_path.exists(): cache_path.unlink() # Load existing cache if cache_path.exists(): cached = pd.read_parquet(cache_path) already_cached = set(cached['Table'].unique()) else: cached = None already_cached = set() # Scrape only missing tables missing = [t for t in tables if t.upper() not in already_cached] new_dfs = [] for table in missing: url = get_sap_table_url(table) df = get_sap_table_structure(url) if df is not None: df['Table'] = table.upper() df['Table Description'] = get_sap_table_description(url) new_dfs.append(df) # Combine and save all_dfs = [d for d in [cached] + new_dfs if d is not None] if not all_dfs: return None result = pd.concat(all_dfs, ignore_index=True) if missing: # only write if something new was added result.to_parquet(cache_path, index=False) # Return only requested tables return result[result['Table'].isin([t.upper() for t in tables])] ``` The key things it does: - Loads the cache and checks what’s already there - Only scrapes the missing tables - Appends new tables and saves back - Returns only the tables you asked for (even if the cache has more) - `force_refresh=True` deletes the cache and starts fresh